Nov 03 15:59:52.272070 ip-10-0-129-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 15:59:52.272085 ip-10-0-129-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 15:59:52.272094 ip-10-0-129-122 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 15:59:52.272423 ip-10-0-129-122 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 16:00:02.350839 ip-10-0-129-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 16:00:02.350854 ip-10-0-129-122 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3500c74ba0ad44e1a5e5a760f2d8b4da -- Nov 03 16:02:03.809335 ip-10-0-129-122 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:02:04.194761 ip-10-0-129-122 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:02:04.194761 ip-10-0-129-122 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:02:04.194761 ip-10-0-129-122 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:02:04.194761 ip-10-0-129-122 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:02:04.199208 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199129 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.199208 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199208 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199213 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199216 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199220 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199223 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199225 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199228 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199231 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199234 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199236 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199239 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199243 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199247 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199250 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199252 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199255 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199257 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199260 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199263 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.199281 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199265 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199271 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199274 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199276 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199279 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199281 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199284 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199287 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199289 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199292 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199296 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199298 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199301 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199303 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199306 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199308 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199310 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199313 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199315 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.199723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199317 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199320 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199322 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199325 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199328 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199330 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199332 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199335 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199337 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199340 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199343 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199345 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199348 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199351 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199353 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199356 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199359 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199361 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199364 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199366 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.200212 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199368 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199371 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199373 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199376 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199378 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199380 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199383 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199386 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199388 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199390 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199393 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199395 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199398 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199400 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199403 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199405 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199408 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199411 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199413 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.200675 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199415 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199418 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199420 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199423 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199425 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199430 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199433 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199435 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199440 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199442 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199445 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199448 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199450 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199453 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199457 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199459 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199461 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199464 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.201143 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199466 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199468 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199471 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199474 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199477 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199479 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.199482 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200260 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200267 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200271 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200274 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200277 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200279 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200284 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200289 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200292 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200295 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200298 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200300 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.201611 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200303 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200306 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200309 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200311 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200315 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200318 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200321 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200323 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200326 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200328 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200331 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200334 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200336 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200339 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200341 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200344 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200346 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200349 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200352 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.202078 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200354 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200357 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200362 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200365 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200368 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200370 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200373 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200375 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200378 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200380 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200383 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200386 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200389 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200391 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200394 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200396 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200399 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200401 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200404 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.202542 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200406 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200409 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200411 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200414 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200416 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200419 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200421 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200424 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200426 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200429 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200431 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200434 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200437 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200439 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200442 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200444 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200447 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200449 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200452 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200455 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.203032 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200457 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200460 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200462 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200465 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200467 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200471 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200473 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200475 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200478 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200481 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200484 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200486 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200489 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200491 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200494 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200496 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200499 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200501 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200504 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200506 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.203538 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200509 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200511 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200514 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200517 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200519 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200522 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200524 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200527 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200529 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200532 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200534 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200539 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200541 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200544 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.200546 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201093 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201099 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201103 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201106 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.204038 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201108 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201113 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201116 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201119 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201122 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201124 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201127 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201130 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201135 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201138 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201141 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201143 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201146 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201149 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201152 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201154 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201157 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201159 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.204575 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201162 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201165 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201167 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201170 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201173 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201176 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201180 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201184 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201187 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201192 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201195 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201197 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201200 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201203 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201206 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201209 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201211 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201214 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.205086 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201216 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201218 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201221 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201223 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201226 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201228 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201231 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201235 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201238 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201240 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201243 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201245 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201247 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201250 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201253 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201256 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201258 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201261 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201263 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.205541 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201266 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201268 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201271 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201273 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201276 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201279 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201281 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201285 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201287 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201289 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201294 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201297 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201299 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201302 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201304 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201307 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201309 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201312 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.206006 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201314 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201317 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201320 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201324 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201326 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201329 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201332 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201334 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201337 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201339 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201342 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201344 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201347 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201350 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201352 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201355 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201357 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201360 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201363 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.206485 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201366 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201369 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201372 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201374 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201376 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.201379 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201463 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201470 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201476 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201482 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201489 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201495 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201502 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201507 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201513 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201516 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201519 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201523 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201526 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201528 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201531 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201534 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201537 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201540 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:02:04.206930 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201544 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201547 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201550 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201554 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201557 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201561 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201564 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201567 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201570 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201574 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201577 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201580 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201583 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201586 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201590 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201593 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201596 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201599 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201602 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201606 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201610 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201614 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201617 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201620 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:02:04.207505 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201623 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201627 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201630 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201633 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201636 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201639 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201643 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201646 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201648 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201651 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201654 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201657 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201662 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201666 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201669 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201672 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201675 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201678 2578 flags.go:64] FLAG: --help="false" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201681 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201684 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201688 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201690 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201694 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:02:04.208159 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201697 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201700 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201702 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201705 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201709 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201712 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201715 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201717 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201720 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201723 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201726 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201728 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201731 2578 flags.go:64] FLAG: --lock-file="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201734 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201737 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201739 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201745 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201747 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201750 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201753 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201756 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201759 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201762 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201765 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201769 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:02:04.208715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201772 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201777 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201780 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201783 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201786 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201788 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201792 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201794 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201797 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201804 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201807 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201811 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201814 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201817 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201820 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201823 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201825 2578 flags.go:64] FLAG: --port="10250" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201828 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201831 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00c270b77ab1b0076" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201834 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201837 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201840 2578 flags.go:64] FLAG: --register-node="true" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201843 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201850 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201852 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201855 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:02:04.209405 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201858 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201862 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201865 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201868 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201871 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201874 2578 flags.go:64] FLAG: --runonce="false" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201876 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201879 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201882 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201885 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201888 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201890 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201893 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201896 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201899 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201902 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201905 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201908 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201911 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201914 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201917 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201922 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201925 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201927 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201932 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:02:04.210216 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201934 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201937 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201940 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201943 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201946 2578 flags.go:64] FLAG: --v="2" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201950 2578 flags.go:64] FLAG: --version="false" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201954 2578 flags.go:64] FLAG: --vmodule="" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201958 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.201961 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202107 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202111 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202114 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202117 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202119 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202122 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202124 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202127 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202129 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202131 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202134 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202136 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.211299 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202139 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202141 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202144 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202147 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202150 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202153 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202156 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202158 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202161 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202165 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202167 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202170 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202172 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202175 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202177 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202181 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202185 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202189 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202192 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.211794 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202195 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202198 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202200 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202203 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202205 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202210 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202213 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202215 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202218 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202220 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202223 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202225 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202228 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202230 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202232 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202235 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202238 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202241 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202244 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.212265 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202246 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202248 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202251 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202255 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202258 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202261 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202264 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202267 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202269 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202272 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202274 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202277 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202279 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202282 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202284 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202286 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202289 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202291 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202294 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202297 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.212716 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202299 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202302 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202304 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202307 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202309 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202312 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202315 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202317 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202319 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202323 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202326 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202328 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202331 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202333 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202335 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202343 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202345 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202348 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202350 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.213234 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202353 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202355 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202358 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202360 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202363 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202365 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202367 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202370 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202372 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202375 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202377 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202380 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202383 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202385 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202390 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.213723 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.202415 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202452 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202457 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202460 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202464 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202468 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202471 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202474 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202476 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202479 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202481 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202484 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202487 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202490 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202493 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202497 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202500 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202502 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.214128 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202505 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202508 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202510 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202513 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202515 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202521 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202524 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202528 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202530 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202533 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202536 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202539 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202542 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202544 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202547 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202549 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202552 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202556 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202558 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.214598 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202561 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202563 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202566 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202568 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202571 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202574 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202577 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202580 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202582 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202584 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202587 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202590 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202592 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202595 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202597 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202600 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202602 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202605 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202607 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202610 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.215094 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202612 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202614 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202618 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202621 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202624 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202626 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202628 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202631 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202633 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202636 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202638 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202641 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202643 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202646 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202648 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202651 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202653 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202657 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202660 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202662 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.215573 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202665 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202667 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202670 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202672 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202674 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202677 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202679 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202682 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202684 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202687 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202689 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202691 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202694 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202696 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202699 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202701 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202704 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202707 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202709 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.216600 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202712 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202714 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202717 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202719 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202722 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202724 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202728 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202731 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.202735 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.210247 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.210264 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.210280 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.210285 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210359 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210364 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210368 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210371 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210374 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210376 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.217088 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210379 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210382 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210384 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210387 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210389 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210392 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210394 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210397 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210400 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210403 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210408 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210411 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210415 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210417 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210420 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210423 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210425 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210428 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210430 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.217539 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210434 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210436 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210439 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210442 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210446 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210449 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210452 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210454 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210457 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210459 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210462 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210464 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210467 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210469 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210472 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210474 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210479 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210482 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210484 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.218000 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210487 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210490 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210493 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210495 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210498 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210500 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210503 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210505 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210508 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210510 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210513 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210515 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210519 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210524 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210527 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210530 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210533 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210536 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210539 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210541 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.218788 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210544 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210547 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210549 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210551 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210554 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210556 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210559 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210561 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210564 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210566 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210569 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210571 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210574 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210576 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210579 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210581 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210584 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210586 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210589 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210591 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.219567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210593 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210596 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210598 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210601 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210603 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210606 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210608 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210611 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210613 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210616 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210619 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210621 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210624 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210626 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210629 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210631 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210634 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210636 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210639 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.220072 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210642 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210644 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.210671 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210846 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210851 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210853 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210856 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210859 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210862 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210866 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210869 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210872 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210875 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210877 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210880 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210882 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.220568 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210885 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210887 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210890 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210892 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210895 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210897 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210900 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210902 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210905 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210908 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210910 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210913 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210915 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210918 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210920 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210922 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210925 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210927 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210930 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.220955 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210932 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210935 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210937 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210940 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210942 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210945 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210947 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210950 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210952 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210955 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210957 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210960 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210962 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210964 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210967 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210969 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210971 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210974 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210977 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.210999 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.221416 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211004 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211007 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211010 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211013 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211015 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211018 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211021 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211023 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211026 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211028 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211031 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211033 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211036 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211039 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211042 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211044 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211046 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211050 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211054 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211058 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.221919 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211062 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211065 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211068 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211071 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211074 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211076 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211079 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211081 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211083 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211086 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211088 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211091 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211093 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211096 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211098 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211101 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211104 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211106 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211109 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211111 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.222701 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211114 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211116 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211118 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211121 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211123 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211126 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211128 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211131 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211133 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211138 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211141 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211143 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211146 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211148 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.211172 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211213 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:02:04.223194 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211217 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211220 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211223 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211226 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211229 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211231 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211236 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211239 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211241 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211243 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211246 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211249 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211252 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211255 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211257 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211260 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211262 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211265 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:02:04.223567 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211267 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211270 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211272 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211275 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211277 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211280 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211282 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211288 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211290 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211293 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211295 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211298 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211301 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211303 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211306 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211308 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211311 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211313 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211318 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211320 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:02:04.224090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211323 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211325 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211328 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211330 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211333 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211335 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211338 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211341 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211343 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211349 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211351 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211353 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211356 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211358 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211361 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211363 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211366 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211368 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:02:04.224577 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211372 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211376 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211379 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211383 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211386 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211388 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211391 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211393 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211398 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211400 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211404 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211406 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211409 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211411 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211414 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211416 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211419 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211421 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:02:04.225206 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211423 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211430 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211433 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211436 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211438 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211440 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211443 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211445 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211448 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211450 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211453 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211455 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211458 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211460 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211465 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211467 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211470 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211472 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211475 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:02:04.225703 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211477 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211479 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211482 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211485 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211490 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211493 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211497 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:04.211500 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.211555 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.213832 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.214771 2578 server.go:1014] "Starting client certificate rotation" Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.214867 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.214899 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.216700 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:02:04.226185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.220199 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:02:04.231268 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.231252 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:02:04.234117 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.234101 2578 log.go:25] "Validated CRI v1 image API" Nov 03 16:02:04.234952 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.234927 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:02:04.239756 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.239728 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a59e7546-0cd1-4458-85aa-7352cb46b66e:/dev/nvme0n1p4 fa59dc6f-986c-4a78-9e81-2e7a0c7d41b6:/dev/nvme0n1p3] Nov 03 16:02:04.239845 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.239755 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 16:02:04.245032 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.244895 2578 manager.go:217] Machine: {Timestamp:2026-11-03 16:02:04.243709461 +0000 UTC m=+0.318170240 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098962 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe9bba1ec7ecacf65149a9fc4a0e1 SystemUUID:ec2fe9bb-a1ec-7eca-cf65-149a9fc4a0e1 BootID:3500c74b-a0ad-44e1-a5e5-a760f2d8b4da Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:85:b9:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:85:b9:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:e6:d1:4e:f5:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:02:04.245032 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.245025 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:02:04.245200 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.245150 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:02:04.247230 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.247202 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:02:04.247397 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.247231 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-122.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:02:04.247532 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.247402 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:02:04.247532 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.247414 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:02:04.247532 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.247457 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:02:04.247532 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.247483 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:02:04.248192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.248178 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:02:04.251388 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251373 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:02:04.251760 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251747 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:02:04.251895 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251886 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:02:04.251924 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251917 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:02:04.251959 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251932 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:02:04.251959 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251946 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:02:04.252060 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251961 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:02:04.252060 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.251973 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:02:04.252317 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.252305 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:02:04.254319 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.254301 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:02:04.258164 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.258148 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:02:04.259391 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259367 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:02:04.259477 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259421 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:02:04.259631 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259618 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:02:04.259659 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259641 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:02:04.259659 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259651 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:02:04.259719 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259660 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:02:04.259719 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259676 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:02:04.259719 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.259686 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:02:04.260209 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260189 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:02:04.260209 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260208 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:02:04.260345 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260221 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:02:04.260345 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260231 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:02:04.260345 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260245 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:02:04.260345 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260259 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:02:04.260859 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260838 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:02:04.260859 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.260861 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:02:04.265363 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.265346 2578 server.go:1263] "Started kubelet" Nov 03 16:02:04.265450 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.265415 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:02:04.265499 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.265433 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:02:04.265544 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.265500 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:02:04.266101 ip-10-0-129-122 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:02:04.266382 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.266349 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:02:04.267258 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.267222 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:02:04.270315 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.270278 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-122.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 16:02:04.270807 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.270790 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:02:04.270887 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.270807 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:02:04.270975 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.270959 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.271446 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.271331 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:02:04.271446 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.271352 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:02:04.271560 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.271454 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:02:04.271560 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.271499 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:02:04.271560 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.271506 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:02:04.272287 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272267 2578 factory.go:55] Registering systemd factory Nov 03 16:02:04.272360 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272291 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:02:04.272862 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272843 2578 factory.go:153] Registering CRI-O factory Nov 03 16:02:04.272862 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272862 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 16:02:04.272976 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272911 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:02:04.272976 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272928 2578 factory.go:103] Registering Raw factory Nov 03 16:02:04.272976 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.272944 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 16:02:04.273119 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.273055 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 16:02:04.273888 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.273834 2578 manager.go:319] Starting recovery of all containers Nov 03 16:02:04.274079 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.274038 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-122.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 16:02:04.274152 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.274128 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-122.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 16:02:04.274375 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.274344 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 16:02:04.274728 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.273749 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-122.us-east-2.compute.internal.18e4946798bc40ae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-122.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-122.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 16:02:04.265316526 +0000 UTC m=+0.339777304,LastTimestamp:2026-11-03 16:02:04.265316526 +0000 UTC m=+0.339777304,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-122.us-east-2.compute.internal,}" Nov 03 16:02:04.277497 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.277431 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s42xs" Nov 03 16:02:04.280536 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.280513 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:02:04.285533 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.284300 2578 manager.go:324] Recovery completed Nov 03 16:02:04.285840 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.285753 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 03 16:02:04.286017 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.285973 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 16:02:04.289450 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.289425 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s42xs" Nov 03 16:02:04.289726 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.289713 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.291746 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.291726 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.291817 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.291761 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.291817 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.291774 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.292376 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.292364 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:02:04.292376 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.292374 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.292498 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.292376 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:02:04.292498 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.292434 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:02:04.293093 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.293077 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.293139 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.293104 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.293139 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.293114 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.294267 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.294252 2578 policy_none.go:50] "Start" Nov 03 16:02:04.294346 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.294277 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:02:04.294346 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.294290 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:02:04.294490 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.294429 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-122.us-east-2.compute.internal.18e494679a4f88af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-122.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-122.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-122.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 16:02:04.291745967 +0000 UTC m=+0.366206746,LastTimestamp:2026-11-03 16:02:04.291745967 +0000 UTC m=+0.366206746,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-122.us-east-2.compute.internal,}" Nov 03 16:02:04.296302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.296290 2578 policy_none.go:44] "Start" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.333731 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336087 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.336113 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336121 2578 server.go:87] "Starting device plugin registration server" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336338 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336350 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336460 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336534 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.336543 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.337205 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.337243 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.346780 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.346800 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.346820 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.346848 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 16:02:04.362256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.350207 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:02:04.436660 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.436639 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.437572 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.437557 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.437654 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.437586 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.437654 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.437600 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.437654 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.437636 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.447266 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.447199 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.447266 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.447219 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-122.us-east-2.compute.internal\": node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.447368 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.447289 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal"] Nov 03 16:02:04.447368 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.447352 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.448124 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.448109 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.448188 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.448137 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.448188 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.448150 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.450247 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.450236 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.450386 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.450373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.450422 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.450401 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.451151 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.451128 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.451151 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.451151 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.451267 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.451160 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.451267 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.451207 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.451267 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.451231 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.451267 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.451248 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.453161 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.453144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.453243 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.453175 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:02:04.453905 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.453879 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:02:04.453905 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.453918 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:02:04.454063 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.453932 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:02:04.472910 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.472889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.473011 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.472915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.473011 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.472932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d723d695ab9475e224b694e4eda0376e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"d723d695ab9475e224b694e4eda0376e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.478558 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.478541 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.484421 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.484402 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.486735 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.486721 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.573321 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.573283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.573321 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.573321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d723d695ab9475e224b694e4eda0376e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"d723d695ab9475e224b694e4eda0376e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.573531 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.573339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.573531 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.573417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.573531 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.573449 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d723d695ab9475e224b694e4eda0376e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"d723d695ab9475e224b694e4eda0376e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.573531 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.573478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.579399 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.579381 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.680143 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.680110 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.780902 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.780841 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.788033 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.788002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.790151 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:04.790137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:04.881689 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.881661 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:04.982186 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:04.982155 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:05.082707 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:05.082642 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:05.183150 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:05.183121 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:05.215306 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.215278 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 16:02:05.215698 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.215446 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:02:05.284242 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:05.284213 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:02:05.293754 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.293716 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:57:04 +0000 UTC" deadline="2028-07-01 16:31:34.911149351 +0000 UTC" Nov 03 16:02:05.293754 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.293752 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14544h29m29.617401704s" Nov 03 16:02:05.316311 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.316295 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:02:05.363321 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.363305 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:02:05.372645 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.372623 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:05.385547 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.385518 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:02:05.386348 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.386325 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:02:05.396540 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.396519 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:02:05.488227 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:05.488192 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd723d695ab9475e224b694e4eda0376e.slice/crio-1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd WatchSource:0}: Error finding container 1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd: Status 404 returned error can't find the container with id 1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd Nov 03 16:02:05.488895 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:05.488875 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd8bc170f67d79edc74165820e4f245.slice/crio-174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b WatchSource:0}: Error finding container 174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b: Status 404 returned error can't find the container with id 174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b Nov 03 16:02:05.491621 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.491602 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:02:05.493551 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:05.493537 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:02:06.202724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.202676 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:02:06.253366 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.253334 2578 apiserver.go:52] "Watching apiserver" Nov 03 16:02:06.262692 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.262659 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:02:06.263169 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.263144 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-n5726","openshift-multus/multus-hpmc5","openshift-multus/network-metrics-daemon-sfkbr","openshift-network-diagnostics/network-check-target-n6lcf","openshift-network-operator/iptables-alerter-9x8lc","kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-lclsg","kube-system/global-pull-secret-syncer-v5zrn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-kjjzr","openshift-ovn-kubernetes/ovnkube-node-s7mkd","kube-system/konnectivity-agent-xmqfv","openshift-dns/node-resolver-tgtnn"] Nov 03 16:02:06.265656 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.265634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.267848 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.267825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.269916 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.269889 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.270100 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.270070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.270177 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.270139 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:06.270177 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.270146 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jdvsk\"" Nov 03 16:02:06.270288 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.270186 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.271923 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.271232 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 16:02:06.271923 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.271748 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:02:06.271923 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.271890 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c56qh\"" Nov 03 16:02:06.271923 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.271895 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:02:06.272256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.272234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:06.272359 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.272306 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:06.274904 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.274890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.277154 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.277126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.280186 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280167 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.280666 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-kubernetes\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.280749 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-conf\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.280749 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-agent-certs\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.280832 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-konnectivity-ca\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.280866 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysconfig\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.280897 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-modprobe-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.280949 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-sys\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281111 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-host\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281111 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.280967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-tmp\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281111 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281111 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-systemd\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281111 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.281332 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47dw2\" (UniqueName: \"kubernetes.io/projected/26139b24-412b-40cd-8247-34995d62bcc9-kube-api-access-47dw2\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.281332 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-lib-modules\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281332 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8ng\" (UniqueName: \"kubernetes.io/projected/9c184456-2ccd-47b7-8b1e-5c410e3248f3-kube-api-access-9b8ng\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281433 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:06.281433 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-run\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281433 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-var-lib-kubelet\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.281537 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.281441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-tuned\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.282861 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.282730 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.284906 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.284883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.285578 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.285558 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:02:06.287265 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.287245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.287359 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.287306 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:06.289715 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.289695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.292203 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.292179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.316510 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.316491 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g5gfq" Nov 03 16:02:06.329748 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.329729 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g5gfq" Nov 03 16:02:06.354372 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.354327 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"2dd8bc170f67d79edc74165820e4f245","Type":"ContainerStarted","Data":"174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b"} Nov 03 16:02:06.355751 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.355730 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"d723d695ab9475e224b694e4eda0376e","Type":"ContainerStarted","Data":"1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd"} Nov 03 16:02:06.372540 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.372500 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:02:06.376730 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.376698 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.376730 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.376717 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2h492\"" Nov 03 16:02:06.376888 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.376776 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:02:06.377172 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.376982 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.377172 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377129 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:02:06.377172 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377135 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.377484 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377463 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.377484 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377480 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:02:06.377659 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377508 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:02:06.377659 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377524 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.377659 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377511 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.377659 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377576 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.377831 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377702 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:02:06.377831 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.377726 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:02:06.378508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.378487 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.378608 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.378508 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.379146 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379096 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:02:06.379146 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379106 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.379146 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379121 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:02:06.379146 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379131 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:02:06.379146 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379146 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qk9pq\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379232 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379248 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dchj7\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379264 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379293 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379306 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5rhh2\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379315 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgwbg\"" Nov 03 16:02:06.379466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.379308 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drdlv\"" Nov 03 16:02:06.380767 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.380748 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r589\"" Nov 03 16:02:06.380895 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.380879 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:02:06.381919 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.381898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-agent-certs\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.381970 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.381935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysconfig\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.381970 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.381960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cnibin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.382058 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.381978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-conf-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.382058 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.381995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-etc-kubernetes\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.382058 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-sys\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.382192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmpj\" (UniqueName: \"kubernetes.io/projected/7128ff3f-3a5a-40c8-98a1-2ec63572519b-kube-api-access-7pmpj\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.382192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysconfig\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.382192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9w66\" (UniqueName: \"kubernetes.io/projected/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-kube-api-access-j9w66\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.382192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-bin\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.382192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dw2\" (UniqueName: \"kubernetes.io/projected/26139b24-412b-40cd-8247-34995d62bcc9-kube-api-access-47dw2\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.382192 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-sys\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.382475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7128ff3f-3a5a-40c8-98a1-2ec63572519b-tmp-dir\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.382475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-log-socket\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.382475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382371 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:02:06.382475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-config\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.382475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovn-node-metrics-cert\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.382475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-var-lib-kubelet\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.382704 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-socket-dir-parent\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.382704 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-dbus\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.382704 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.382704 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-var-lib-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.382704 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-conf\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.382704 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-konnectivity-ca\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-sys-fs\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-host\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-k8s-cni-cncf-io\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-conf\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-node-log\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-tmp\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8ds\" (UniqueName: \"kubernetes.io/projected/6567421d-b34c-4a99-8fad-c314c9538374-kube-api-access-tf8ds\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-env-overrides\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-systemd\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.382988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-lib-modules\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383003 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6567421d-b34c-4a99-8fad-c314c9538374-iptables-alerter-script\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-systemd\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-var-lib-kubelet\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.383214 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.383294 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:06.883255694 +0000 UTC m=+2.957716479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-konnectivity-ca\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-lib-modules\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-run\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-socket-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cni-binary-copy\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.383655 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-run\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-bin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zhv\" (UniqueName: \"kubernetes.io/projected/b2396d77-c56c-48da-96f1-08b0287dab5f-kube-api-access-h9zhv\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-hostroot\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.383906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-daemon-config\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-etc-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.384222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-device-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7128ff3f-3a5a-40c8-98a1-2ec63572519b-hosts-file\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-multus-certs\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8ng\" (UniqueName: \"kubernetes.io/projected/9c184456-2ccd-47b7-8b1e-5c410e3248f3-kube-api-access-9b8ng\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-registration-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-serviceca\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.384463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfn7\" (UniqueName: \"kubernetes.io/projected/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-kube-api-access-ttfn7\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.384662 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-system-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.384662 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-kubelet-config\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.384662 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cnibin\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.384662 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-slash\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384662 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:06.384662 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.384833 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.384833 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-kubelet\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384833 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-ovn\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384833 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384962 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq64w\" (UniqueName: \"kubernetes.io/projected/0533b853-c8d6-4018-bc6c-20f99c17f1ef-kube-api-access-fq64w\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.384962 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-netd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384962 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwl9d\" (UniqueName: \"kubernetes.io/projected/63f2d5ae-3734-4546-9f13-8ec3b049a83e-kube-api-access-pwl9d\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.384962 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-modprobe-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.384962 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-host\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.384962 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-netns\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.385218 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.384982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.385218 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.385218 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.385218 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-netns\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.385218 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-systemd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.385218 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385165 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-tuned\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-multus\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-kubelet\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-systemd-units\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-script-lib\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6567421d-b34c-4a99-8fad-c314c9538374-host-slash\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.385508 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-kubernetes\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.385720 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-kubernetes\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.386021 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.385998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-host\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.386093 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.386014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-modprobe-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.386158 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.386128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.386213 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.386193 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-tmp\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.386499 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.386461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-agent-certs\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.388839 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.388814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-tuned\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.398379 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.398321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dw2\" (UniqueName: \"kubernetes.io/projected/26139b24-412b-40cd-8247-34995d62bcc9-kube-api-access-47dw2\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.398514 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.398494 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:06.398569 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.398524 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:06.398606 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.398569 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:06.398662 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.398650 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:02:06.898616598 +0000 UTC m=+2.973077383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:06.400851 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.400806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8ng\" (UniqueName: \"kubernetes.io/projected/9c184456-2ccd-47b7-8b1e-5c410e3248f3-kube-api-access-9b8ng\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.485851 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.485755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8ds\" (UniqueName: \"kubernetes.io/projected/6567421d-b34c-4a99-8fad-c314c9538374-kube-api-access-tf8ds\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.485851 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.485798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486131 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.485947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486131 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.485979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-env-overrides\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.486131 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6567421d-b34c-4a99-8fad-c314c9538374-iptables-alerter-script\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.486131 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.486131 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-socket-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486404 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-socket-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486404 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.486404 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cni-binary-copy\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486404 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-bin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486404 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zhv\" (UniqueName: \"kubernetes.io/projected/b2396d77-c56c-48da-96f1-08b0287dab5f-kube-api-access-h9zhv\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-hostroot\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-daemon-config\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-etc-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-device-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7128ff3f-3a5a-40c8-98a1-2ec63572519b-hosts-file\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-multus-certs\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-registration-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-serviceca\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfn7\" (UniqueName: \"kubernetes.io/projected/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-kube-api-access-ttfn7\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-system-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.486724 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-kubelet-config\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cnibin\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6567421d-b34c-4a99-8fad-c314c9538374-iptables-alerter-script\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-slash\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-hostroot\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-device-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-kubelet\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-ovn\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.486900 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq64w\" (UniqueName: \"kubernetes.io/projected/0533b853-c8d6-4018-bc6c-20f99c17f1ef-kube-api-access-fq64w\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-netd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.486959 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:02:06.986944026 +0000 UTC m=+3.061404793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwl9d\" (UniqueName: \"kubernetes.io/projected/63f2d5ae-3734-4546-9f13-8ec3b049a83e-kube-api-access-pwl9d\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-netns\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.487504 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-netns\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-systemd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7128ff3f-3a5a-40c8-98a1-2ec63572519b-hosts-file\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-multus\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-kubelet\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-systemd-units\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-script-lib\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6567421d-b34c-4a99-8fad-c314c9538374-host-slash\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cnibin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487404 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cni-binary-copy\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-conf-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-etc-kubernetes\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-bin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.488463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmpj\" (UniqueName: \"kubernetes.io/projected/7128ff3f-3a5a-40c8-98a1-2ec63572519b-kube-api-access-7pmpj\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w66\" (UniqueName: \"kubernetes.io/projected/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-kube-api-access-j9w66\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-bin\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7128ff3f-3a5a-40c8-98a1-2ec63572519b-tmp-dir\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-log-socket\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-config\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovn-node-metrics-cert\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-socket-dir-parent\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-dbus\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-serviceca\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-var-lib-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-sys-fs\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-host\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-k8s-cni-cncf-io\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-node-log\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-node-log\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.489365 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-system-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-netns\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cnibin\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.486729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-env-overrides\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-kubelet-config\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-slash\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-systemd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488237 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-multus\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-kubelet\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.487288 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-multus-certs\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-etc-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-registration-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-systemd-units\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-daemon-config\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.490194 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488611 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-kubelet\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488698 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-netd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-script-lib\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.488979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-netns\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-dbus\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489351 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-var-lib-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489415 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-config\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6567421d-b34c-4a99-8fad-c314c9538374-host-slash\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489673 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-bin\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cnibin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489857 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-conf-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-sys-fs\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7128ff3f-3a5a-40c8-98a1-2ec63572519b-tmp-dir\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489909 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-host\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.490948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489931 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-log-socket\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.491927 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-etc-kubernetes\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.491927 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-k8s-cni-cncf-io\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.491927 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.489958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-ovn\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.491927 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.490025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-socket-dir-parent\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.491927 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.490231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.491927 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.491580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovn-node-metrics-cert\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.501157 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.501108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8ds\" (UniqueName: \"kubernetes.io/projected/6567421d-b34c-4a99-8fad-c314c9538374-kube-api-access-tf8ds\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.502467 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.502441 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfn7\" (UniqueName: \"kubernetes.io/projected/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-kube-api-access-ttfn7\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.503532 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.503502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zhv\" (UniqueName: \"kubernetes.io/projected/b2396d77-c56c-48da-96f1-08b0287dab5f-kube-api-access-h9zhv\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.503844 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.503822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmpj\" (UniqueName: \"kubernetes.io/projected/7128ff3f-3a5a-40c8-98a1-2ec63572519b-kube-api-access-7pmpj\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.504821 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.504800 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9w66\" (UniqueName: \"kubernetes.io/projected/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-kube-api-access-j9w66\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.508342 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.508316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwl9d\" (UniqueName: \"kubernetes.io/projected/63f2d5ae-3734-4546-9f13-8ec3b049a83e-kube-api-access-pwl9d\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.508445 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.508420 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq64w\" (UniqueName: \"kubernetes.io/projected/0533b853-c8d6-4018-bc6c-20f99c17f1ef-kube-api-access-fq64w\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.579263 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.579228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:02:06.586457 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.586435 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:06.588532 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.588505 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c184456_2ccd_47b7_8b1e_5c410e3248f3.slice/crio-968957b011969c3708e15f2388b63d6c986e73bffa01d21362061a2f1043fb2e WatchSource:0}: Error finding container 968957b011969c3708e15f2388b63d6c986e73bffa01d21362061a2f1043fb2e: Status 404 returned error can't find the container with id 968957b011969c3708e15f2388b63d6c986e73bffa01d21362061a2f1043fb2e Nov 03 16:02:06.596575 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.596549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:02:06.596902 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.596874 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eab8211_6bb8_4c65_a1a1_f2d0474952a2.slice/crio-bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89 WatchSource:0}: Error finding container bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89: Status 404 returned error can't find the container with id bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89 Nov 03 16:02:06.604252 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.604231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:02:06.608429 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.608177 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6567421d_b34c_4a99_8fad_c314c9538374.slice/crio-8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3 WatchSource:0}: Error finding container 8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3: Status 404 returned error can't find the container with id 8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3 Nov 03 16:02:06.616582 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.616559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:02:06.619914 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.619889 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5726" Nov 03 16:02:06.620245 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.620218 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7128ff3f_3a5a_40c8_98a1_2ec63572519b.slice/crio-b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1 WatchSource:0}: Error finding container b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1: Status 404 returned error can't find the container with id b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1 Nov 03 16:02:06.628055 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.627887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hpmc5" Nov 03 16:02:06.628337 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.628312 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2396d77_c56c_48da_96f1_08b0287dab5f.slice/crio-f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8 WatchSource:0}: Error finding container f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8: Status 404 returned error can't find the container with id f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8 Nov 03 16:02:06.630361 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.630339 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7b62ab_f1df_47f3_99b3_b34efd73a23e.slice/crio-86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601 WatchSource:0}: Error finding container 86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601: Status 404 returned error can't find the container with id 86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601 Nov 03 16:02:06.639087 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.639064 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0533b853_c8d6_4018_bc6c_20f99c17f1ef.slice/crio-d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61 WatchSource:0}: Error finding container d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61: Status 404 returned error can't find the container with id d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61 Nov 03 16:02:06.663152 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.663130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:02:06.673926 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.673889 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30abbfa0_1328_4696_b0c7_7e5dd3947fa7.slice/crio-29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493 WatchSource:0}: Error finding container 29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493: Status 404 returned error can't find the container with id 29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493 Nov 03 16:02:06.675935 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.675912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:06.685595 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:06.685567 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f2d5ae_3734_4546_9f13_8ec3b049a83e.slice/crio-af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f WatchSource:0}: Error finding container af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f: Status 404 returned error can't find the container with id af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f Nov 03 16:02:06.892313 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.892218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:06.892479 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.892358 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:06.892479 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.892448 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.892424874 +0000 UTC m=+3.966885658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:06.993463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.993424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:06.993463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:06.993460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:06.993704 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.993587 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:06.993704 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.993618 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:06.993704 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.993642 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:06.993704 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.993644 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.993624118 +0000 UTC m=+4.068084885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:06.993704 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.993658 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:06.993909 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:06.993716 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:02:07.993698754 +0000 UTC m=+4.068159536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:07.334470 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.334164 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:57:06 +0000 UTC" deadline="2028-06-03 03:48:12.360512634 +0000 UTC" Nov 03 16:02:07.334470 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.334233 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13859h46m5.026283222s" Nov 03 16:02:07.369779 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.369741 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerStarted","Data":"29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493"} Nov 03 16:02:07.377608 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.377525 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hpmc5" event={"ID":"0533b853-c8d6-4018-bc6c-20f99c17f1ef","Type":"ContainerStarted","Data":"d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61"} Nov 03 16:02:07.383309 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.382963 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5726" event={"ID":"7f7b62ab-f1df-47f3-99b3-b34efd73a23e","Type":"ContainerStarted","Data":"86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601"} Nov 03 16:02:07.385830 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.385190 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" event={"ID":"b2396d77-c56c-48da-96f1-08b0287dab5f","Type":"ContainerStarted","Data":"f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8"} Nov 03 16:02:07.386897 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.386867 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgtnn" event={"ID":"7128ff3f-3a5a-40c8-98a1-2ec63572519b","Type":"ContainerStarted","Data":"b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1"} Nov 03 16:02:07.389272 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.389150 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f"} Nov 03 16:02:07.395101 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.395074 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x8lc" event={"ID":"6567421d-b34c-4a99-8fad-c314c9538374","Type":"ContainerStarted","Data":"8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3"} Nov 03 16:02:07.402961 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.402917 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xmqfv" event={"ID":"3eab8211-6bb8-4c65-a1a1-f2d0474952a2","Type":"ContainerStarted","Data":"bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89"} Nov 03 16:02:07.426728 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.426696 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lclsg" event={"ID":"9c184456-2ccd-47b7-8b1e-5c410e3248f3","Type":"ContainerStarted","Data":"968957b011969c3708e15f2388b63d6c986e73bffa01d21362061a2f1043fb2e"} Nov 03 16:02:07.902069 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:07.901764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:07.902227 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:07.902151 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:07.902293 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:07.902232 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:09.902211134 +0000 UTC m=+5.976671917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:08.002770 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.002679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:08.002770 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.002727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:08.002981 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.002867 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:08.002981 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.002929 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.002911238 +0000 UTC m=+6.077372019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:08.003139 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.003016 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:08.003608 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.003473 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:08.003608 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.003508 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:08.003608 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.003576 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:02:10.003548607 +0000 UTC m=+6.078009388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:08.335541 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.335442 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:57:06 +0000 UTC" deadline="2028-07-06 14:19:27.736371285 +0000 UTC" Nov 03 16:02:08.335541 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.335489 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14662h17m19.40088714s" Nov 03 16:02:08.354928 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.354900 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:08.355118 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.355045 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:08.355522 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.355503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:08.355637 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.355617 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:08.355721 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.355706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:08.355833 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:08.355811 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:08.445260 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.445222 2578 generic.go:356] "Generic (PLEG): container finished" podID="2dd8bc170f67d79edc74165820e4f245" containerID="69ae0eb4010d371dcda55d3a0266514ef49066a70c5da055ae86f0d05fd14f6b" exitCode=0 Nov 03 16:02:08.445413 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:08.445285 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"2dd8bc170f67d79edc74165820e4f245","Type":"ContainerDied","Data":"69ae0eb4010d371dcda55d3a0266514ef49066a70c5da055ae86f0d05fd14f6b"} Nov 03 16:02:09.464256 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:09.464222 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"2dd8bc170f67d79edc74165820e4f245","Type":"ContainerStarted","Data":"823ab6a9be1c664db9b45b01caa728e8754b2461c412d8ad9a7a6e48224af087"} Nov 03 16:02:09.924974 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:09.924941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:09.925135 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:09.925115 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:09.925201 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:09.925177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:13.925156895 +0000 UTC m=+9.999617677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:10.025986 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.025903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:10.025986 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.025955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:10.026235 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.026105 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:10.026235 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.026173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:02:14.026152995 +0000 UTC m=+10.100613777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:10.026769 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.026594 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:10.026769 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.026624 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:10.026769 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.026637 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:10.026769 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.026684 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:02:14.026667924 +0000 UTC m=+10.101128699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:10.349376 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.349347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:10.349541 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.349454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:10.349740 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.349718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:10.349847 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.349810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:10.349911 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.349858 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:10.349911 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:10.349904 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:10.471639 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.471602 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"d723d695ab9475e224b694e4eda0376e","Type":"ContainerStarted","Data":"bbd64df2fd40dfcd4cea1aca1c6639b7d4f328bc8a63b908734476e4ceb1fe2d"} Nov 03 16:02:10.474435 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.474409 2578 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="e88bdbc34ae7e51ac8e013a000b5f6c64103337181d811a216997bc737d1a793" exitCode=0 Nov 03 16:02:10.474555 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.474454 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerDied","Data":"e88bdbc34ae7e51ac8e013a000b5f6c64103337181d811a216997bc737d1a793"} Nov 03 16:02:10.491864 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.491791 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" podStartSLOduration=5.491774526 podStartE2EDuration="5.491774526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:10.491405152 +0000 UTC m=+6.565865940" watchObservedRunningTime="2026-11-03 16:02:10.491774526 +0000 UTC m=+6.566235315" Nov 03 16:02:10.492141 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:10.492109 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" podStartSLOduration=5.492101147 podStartE2EDuration="5.492101147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:09.496807391 +0000 UTC m=+5.571268183" watchObservedRunningTime="2026-11-03 16:02:10.492101147 +0000 UTC m=+6.566561936" Nov 03 16:02:11.484220 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:11.484127 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xmqfv" event={"ID":"3eab8211-6bb8-4c65-a1a1-f2d0474952a2","Type":"ContainerStarted","Data":"f6865a7597b0a412f6cf360a38de9ae08b68477205f4b1106dbce36ef879c49e"} Nov 03 16:02:11.819319 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:11.819235 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:11.820224 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:11.819959 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:11.838968 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:11.838574 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xmqfv" podStartSLOduration=3.403928341 podStartE2EDuration="7.838556169s" totalImagesPullingTime="4.434627828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:11.505359409 +0000 UTC m=+7.579820210" watchObservedRunningTime="2026-11-03 16:02:11.838556169 +0000 UTC m=+7.913016955" Nov 03 16:02:12.350076 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:12.350043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:12.350189 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:12.350162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:12.350551 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:12.350532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:12.350667 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:12.350644 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:12.350737 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:12.350711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:12.350793 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:12.350777 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:12.488700 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:12.488626 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5726" event={"ID":"7f7b62ab-f1df-47f3-99b3-b34efd73a23e","Type":"ContainerStarted","Data":"ef40403f4e97dc7ae16b43a8b1863ed9ca5869af16f40428d590dd57677e3330"} Nov 03 16:02:13.493466 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:13.493431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" event={"ID":"b2396d77-c56c-48da-96f1-08b0287dab5f","Type":"ContainerStarted","Data":"515b9f70e1b8e9ad184d734113f126784c1380a2d77ff17b105f550b27a6ebbb"} Nov 03 16:02:13.493949 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:13.493539 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:02:13.956720 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:13.956643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:13.956878 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:13.956821 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:13.956976 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:13.956960 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:21.956918153 +0000 UTC m=+18.031378933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:14.057289 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:14.057249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:14.057439 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:14.057298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:14.057439 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.057429 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:14.057544 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.057490 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:02:22.057473375 +0000 UTC m=+18.131934155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:14.057974 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.057869 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:14.057974 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.057894 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:14.057974 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.057905 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:14.057974 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.057938 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:02:22.057928972 +0000 UTC m=+18.132389751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:14.349786 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:14.349720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:14.349920 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.349799 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:14.350287 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:14.350083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:14.350287 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.350221 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:14.350287 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:14.350274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:14.350535 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:14.350371 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:14.769385 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:14.769332 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:02:15.343128 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.343016 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:02:14.769354769Z","UUID":"24286cfe-64cf-4000-b4be-04f6180f2f79","Handler":null,"Name":"","Endpoint":""} Nov 03 16:02:15.344976 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.344781 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:02:15.345109 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.345031 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:02:15.502680 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.502034 2578 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="3849d2bc5b45110716bf9d6b3878914ede1c930422fe44a0e43e89c20c4b539d" exitCode=0 Nov 03 16:02:15.502680 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.502167 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerDied","Data":"3849d2bc5b45110716bf9d6b3878914ede1c930422fe44a0e43e89c20c4b539d"} Nov 03 16:02:15.505167 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.504715 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hpmc5" event={"ID":"0533b853-c8d6-4018-bc6c-20f99c17f1ef","Type":"ContainerStarted","Data":"4ea0aab5949170aeae8c3ed00f2a54c261555cccedd01c16d445a9f802c0c956"} Nov 03 16:02:15.507646 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.507582 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" event={"ID":"b2396d77-c56c-48da-96f1-08b0287dab5f","Type":"ContainerStarted","Data":"7044e23d5be187052fbd208b393b6ef7ee6d723ff9890931bf50f673ba31eae7"} Nov 03 16:02:15.510163 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.509816 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgtnn" event={"ID":"7128ff3f-3a5a-40c8-98a1-2ec63572519b","Type":"ContainerStarted","Data":"ebf09d1f56719b2a97a86f85986eaf65327a1696df4f9944b897de0362efe6b1"} Nov 03 16:02:15.546807 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.546763 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n5726" podStartSLOduration=6.047995105 podStartE2EDuration="11.54674779s" totalImagesPullingTime="5.498752685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:12.510464026 +0000 UTC m=+8.584924818" watchObservedRunningTime="2026-11-03 16:02:15.54674779 +0000 UTC m=+11.621208577" Nov 03 16:02:15.572967 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.572919 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tgtnn" podStartSLOduration=3.05159943 podStartE2EDuration="11.572904706s" totalImagesPullingTime="8.521305276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:15.572252284 +0000 UTC m=+11.646713074" watchObservedRunningTime="2026-11-03 16:02:15.572904706 +0000 UTC m=+11.647365496" Nov 03 16:02:15.594231 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:15.593676 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hpmc5" podStartSLOduration=3.3351663240000002 podStartE2EDuration="11.593659099s" totalImagesPullingTime="8.258492775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:15.593214558 +0000 UTC m=+11.667675346" watchObservedRunningTime="2026-11-03 16:02:15.593659099 +0000 UTC m=+11.668119919" Nov 03 16:02:16.348482 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:16.347932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:16.348482 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:16.347932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:16.348482 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:16.348178 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:16.348482 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:16.348205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:16.348482 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:16.348332 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:16.348482 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:16.348414 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:16.515114 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:16.515079 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" event={"ID":"b2396d77-c56c-48da-96f1-08b0287dab5f","Type":"ContainerStarted","Data":"e9ccfa926a994057866b9a14d154c1273dd51ca89fe18fdf0c9028d912f338e6"} Nov 03 16:02:16.517325 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:16.517301 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x8lc" event={"ID":"6567421d-b34c-4a99-8fad-c314c9538374","Type":"ContainerStarted","Data":"ffef26d6cfc32270ea67041343a6d0c5974ed2da822dee442f03bf0adb5dfcbe"} Nov 03 16:02:16.544849 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:16.544801 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" podStartSLOduration=3.809253993 podStartE2EDuration="12.544789726s" totalImagesPullingTime="8.735535733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:16.544287855 +0000 UTC m=+12.618748639" watchObservedRunningTime="2026-11-03 16:02:16.544789726 +0000 UTC m=+12.619250513" Nov 03 16:02:17.522006 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:17.521951 2578 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="0b27bc782382af3e7222d0fbf8fb2d563f61485dd8a499bd15faa6428dbde655" exitCode=0 Nov 03 16:02:17.522601 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:17.522081 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerDied","Data":"0b27bc782382af3e7222d0fbf8fb2d563f61485dd8a499bd15faa6428dbde655"} Nov 03 16:02:17.524156 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:17.524126 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lclsg" event={"ID":"9c184456-2ccd-47b7-8b1e-5c410e3248f3","Type":"ContainerStarted","Data":"1f9b7d2c72a4afeb43870093087e3acbb9865aba7458227e91c09f57f245d919"} Nov 03 16:02:17.553199 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:17.553148 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9x8lc" podStartSLOduration=5.270307334 podStartE2EDuration="13.553133179s" totalImagesPullingTime="8.282825845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:16.565167457 +0000 UTC m=+12.639628246" watchObservedRunningTime="2026-11-03 16:02:17.553133179 +0000 UTC m=+13.627593969" Nov 03 16:02:17.573228 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:17.573173 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lclsg" podStartSLOduration=2.821138627 podStartE2EDuration="13.573154738s" totalImagesPullingTime="10.752016111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:17.572885814 +0000 UTC m=+13.647346618" watchObservedRunningTime="2026-11-03 16:02:17.573154738 +0000 UTC m=+13.647615527" Nov 03 16:02:17.788244 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:17.788178 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgtnn_7128ff3f-3a5a-40c8-98a1-2ec63572519b/dns-node-resolver/0.log" Nov 03 16:02:18.347318 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:18.347245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:18.347318 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:18.347253 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:18.347568 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:18.347419 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:18.347568 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:18.347419 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:18.347568 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:18.347536 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:18.347698 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:18.347616 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:18.760020 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:18.759972 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5726_7f7b62ab-f1df-47f3-99b3-b34efd73a23e/node-ca/0.log" Nov 03 16:02:19.530727 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:19.530695 2578 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="f3345b24faa1c66b2aac1cebea988d41c5b63b9ed621a533092a9ba40bf4e1e6" exitCode=0 Nov 03 16:02:19.530877 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:19.530753 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerDied","Data":"f3345b24faa1c66b2aac1cebea988d41c5b63b9ed621a533092a9ba40bf4e1e6"} Nov 03 16:02:19.761572 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:19.761542 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:19.762086 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:19.761684 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:02:19.762378 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:19.762354 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:02:20.347245 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:20.347008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:20.347408 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:20.347351 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:20.347760 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:20.347732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:20.347891 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:20.347851 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:20.347948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:20.347905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:20.348018 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:20.347971 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:22.019434 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.019260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:22.019805 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.019404 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:22.019805 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.019552 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:02:38.019538416 +0000 UTC m=+34.093999180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:22.120200 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.120121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:22.120200 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.120155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:22.120363 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.120275 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:22.120363 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.120279 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:22.120363 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.120298 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:22.120363 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.120308 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:22.120363 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.120331 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:02:38.120315304 +0000 UTC m=+34.194776090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:22.120363 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.120347 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:02:38.120335896 +0000 UTC m=+34.194796675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:22.350743 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.350709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:22.350901 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.350780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:22.350901 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.350803 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:22.351006 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.350891 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:22.351006 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.350930 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:22.351109 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:22.351042 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:22.540279 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.540186 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"c080813c793404f3c35ddbbeb0ef904a44742b4b1fcb1a3a63f55c994505f84e"} Nov 03 16:02:22.540279 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.540257 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"69672dc467384799eb0eb36c532bd927863aa35981ea1223fafc12041e68b383"} Nov 03 16:02:22.540453 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.540295 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"0451c3f4a5c8e3a2103ef65d5795508bdfb30036ddfdcbc2ff80055d4b5fc8d5"} Nov 03 16:02:22.540453 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.540325 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"ad7e49a14764feec82061b1e51840512504514d574559100a2fff58a42de6af2"} Nov 03 16:02:22.540453 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.540352 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"0bb297bed0052711f61141941b8be602d6cb64931049a2eccb3765f8c267ba73"} Nov 03 16:02:22.540453 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.540386 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"fa957ec65d022ba8c25bdbf447c576f3096a45c46aa77f8392eb68a2078345e8"} Nov 03 16:02:22.542823 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.542795 2578 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="e42c5d2c0b979ad11a102797dafb1f710f4d1f9a0eb84616b1c5a27449b33990" exitCode=0 Nov 03 16:02:22.542925 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:22.542834 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerDied","Data":"e42c5d2c0b979ad11a102797dafb1f710f4d1f9a0eb84616b1c5a27449b33990"} Nov 03 16:02:23.547847 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:23.547818 2578 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="7fc80880bea75f2a3a12de836909d721ccbd0d0789a3e3ccedc46142be56d843" exitCode=0 Nov 03 16:02:23.548380 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:23.547871 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerDied","Data":"7fc80880bea75f2a3a12de836909d721ccbd0d0789a3e3ccedc46142be56d843"} Nov 03 16:02:24.349748 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:24.349675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:24.349748 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:24.349705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:24.349934 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:24.349675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:24.349934 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:24.349773 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:24.349934 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:24.349867 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:24.350059 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:24.349938 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:24.552949 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:24.552913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"9cca74c0fcfc68ccbdc97688ac497e7b0646c519cafdc6dbcacd4433e344f3cb"} Nov 03 16:02:24.556117 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:24.556091 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" event={"ID":"30abbfa0-1328-4696-b0c7-7e5dd3947fa7","Type":"ContainerStarted","Data":"215ccbcedd4a477a03bbc624b0873171ec497e0be11caff969d6eec597010f90"} Nov 03 16:02:24.582300 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:24.582260 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kjjzr" podStartSLOduration=8.716651242 podStartE2EDuration="20.582249215s" totalImagesPullingTime="11.865597973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:24.582001197 +0000 UTC m=+20.656461973" watchObservedRunningTime="2026-11-03 16:02:24.582249215 +0000 UTC m=+20.656709998" Nov 03 16:02:26.347134 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.347101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:26.347709 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.347162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:26.347709 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.347170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:26.347709 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:26.347259 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:26.347709 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:26.347356 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:26.347709 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:26.347461 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:26.563912 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.563823 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" event={"ID":"63f2d5ae-3734-4546-9f13-8ec3b049a83e","Type":"ContainerStarted","Data":"6f252d0181831794cd23a08a05f017240331ec3837321390b494ac73e858461d"} Nov 03 16:02:26.564158 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.564143 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:26.564240 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.564205 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:26.580449 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.580427 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:26.600717 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:26.600679 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" podStartSLOduration=7.650249094 podStartE2EDuration="22.600668677s" totalImagesPullingTime="14.950419583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:26.600142619 +0000 UTC m=+22.674603406" watchObservedRunningTime="2026-11-03 16:02:26.600668677 +0000 UTC m=+22.675129464" Nov 03 16:02:27.566618 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:27.566584 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:27.581776 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:27.581756 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:02:28.348193 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:28.348157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:28.348342 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:28.348171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:28.348342 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:28.348291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:28.348342 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:28.348295 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:28.348469 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:28.348363 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:28.348506 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:28.348453 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:30.347497 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:30.347328 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:30.347827 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:30.347335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:30.347827 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:30.347353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:30.347827 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:30.347684 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:30.347827 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:30.347710 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:30.347827 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:30.347565 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:32.347075 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:32.347040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:32.347567 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:32.347057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:32.347567 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:32.347165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:32.347567 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:32.347233 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:32.347567 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:32.347253 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:32.347567 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:32.347334 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:34.349577 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.349549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:34.350151 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.349622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:34.350278 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:34.350255 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:34.350384 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:34.350360 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:34.350588 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.349655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:34.350686 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:34.350658 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:34.997917 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.997872 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sfkbr"] Nov 03 16:02:34.998101 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.998069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:34.998192 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:34.998165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:34.998700 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.998680 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6lcf"] Nov 03 16:02:34.998813 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.998762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:34.998870 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:34.998827 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:34.999295 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.999270 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5zrn"] Nov 03 16:02:34.999397 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:34.999357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:34.999454 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:34.999430 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:36.347821 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:36.347791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:36.348298 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:36.347902 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:36.685507 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:36.685424 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgtnn_7128ff3f-3a5a-40c8-98a1-2ec63572519b/dns-node-resolver/0.log" Nov 03 16:02:37.347349 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:37.347316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:37.347512 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:37.347317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:37.347512 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:37.347451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:37.347512 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:37.347491 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:37.883730 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:37.883700 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5726_7f7b62ab-f1df-47f3-99b3-b34efd73a23e/node-ca/0.log" Nov 03 16:02:38.040267 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:38.040236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:38.040409 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.040363 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:38.040452 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.040418 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs podName:26139b24-412b-40cd-8247-34995d62bcc9 nodeName:}" failed. No retries permitted until 2026-11-03 16:03:10.040403408 +0000 UTC m=+66.114864178 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs") pod "network-metrics-daemon-sfkbr" (UID: "26139b24-412b-40cd-8247-34995d62bcc9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:02:38.140699 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:38.140632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:38.140699 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:38.140662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:38.140851 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.140767 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:38.140851 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.140786 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:02:38.140851 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.140803 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:02:38.140851 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.140813 2578 projected.go:196] Error preparing data for projected volume kube-api-access-bvjww for pod openshift-network-diagnostics/network-check-target-n6lcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:38.140851 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.140814 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret podName:6df99de4-80b2-4cde-b4e8-aa117c1d00de nodeName:}" failed. No retries permitted until 2026-11-03 16:03:10.140801848 +0000 UTC m=+66.215262614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret") pod "global-pull-secret-syncer-v5zrn" (UID: "6df99de4-80b2-4cde-b4e8-aa117c1d00de") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:02:38.141063 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.140864 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww podName:216e3f72-2de8-4789-9f51-a8f13a33cd0c nodeName:}" failed. No retries permitted until 2026-11-03 16:03:10.140849457 +0000 UTC m=+66.215310223 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvjww" (UniqueName: "kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww") pod "network-check-target-n6lcf" (UID: "216e3f72-2de8-4789-9f51-a8f13a33cd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:02:38.347054 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:38.347026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:38.347197 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:38.347126 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6lcf" podUID="216e3f72-2de8-4789-9f51-a8f13a33cd0c" Nov 03 16:02:39.347835 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.347774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:39.347835 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.347801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:39.348271 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:39.347903 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfkbr" podUID="26139b24-412b-40cd-8247-34995d62bcc9" Nov 03 16:02:39.348271 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:02:39.348017 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5zrn" podUID="6df99de4-80b2-4cde-b4e8-aa117c1d00de" Nov 03 16:02:39.629783 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.629706 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeReady" Nov 03 16:02:39.629945 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.629854 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 16:02:39.682023 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.681972 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6tc5s"] Nov 03 16:02:39.734258 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.734225 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4q4t6"] Nov 03 16:02:39.734419 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.734357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.739083 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.739060 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:02:39.739202 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.739116 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:02:39.739411 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.739398 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jk68z\"" Nov 03 16:02:39.739492 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.739481 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:02:39.749564 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.749547 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tc5s"] Nov 03 16:02:39.749649 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.749575 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q4t6"] Nov 03 16:02:39.749649 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.749596 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jksqm"] Nov 03 16:02:39.749712 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.749659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.763649 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.763632 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jksqm"] Nov 03 16:02:39.763734 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.763723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.839302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839281 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:02:39.839410 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839308 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:02:39.839494 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839478 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:02:39.839543 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839533 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:02:39.839600 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839588 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:02:39.839600 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839596 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bb2h\"" Nov 03 16:02:39.839710 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839663 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:02:39.839710 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.839708 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8szjr\"" Nov 03 16:02:39.853859 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.853840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee13f98-11c4-4b25-87e1-f0c4729750f9-tmp-dir\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.853934 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.853865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpvk\" (UniqueName: \"kubernetes.io/projected/1ee13f98-11c4-4b25-87e1-f0c4729750f9-kube-api-access-5kpvk\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.853934 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.853902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jhm\" (UniqueName: \"kubernetes.io/projected/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-api-access-w4jhm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.853934 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.853922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee13f98-11c4-4b25-87e1-f0c4729750f9-config-volume\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.854115 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-data-volume\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.854169 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e678d92f-19e7-49f2-b876-774d845ac141-cert\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.854169 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxg5\" (UniqueName: \"kubernetes.io/projected/e678d92f-19e7-49f2-b876-774d845ac141-kube-api-access-qjxg5\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.854238 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ee13f98-11c4-4b25-87e1-f0c4729750f9-metrics-tls\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.854238 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-crio-socket\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.854295 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.854295 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.854275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955180 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jhm\" (UniqueName: \"kubernetes.io/projected/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-api-access-w4jhm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955180 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee13f98-11c4-4b25-87e1-f0c4729750f9-config-volume\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.955302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-data-volume\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e678d92f-19e7-49f2-b876-774d845ac141-cert\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.955302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxg5\" (UniqueName: \"kubernetes.io/projected/e678d92f-19e7-49f2-b876-774d845ac141-kube-api-access-qjxg5\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.955302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ee13f98-11c4-4b25-87e1-f0c4729750f9-metrics-tls\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.955513 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-crio-socket\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955513 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955513 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955513 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee13f98-11c4-4b25-87e1-f0c4729750f9-tmp-dir\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.955693 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpvk\" (UniqueName: \"kubernetes.io/projected/1ee13f98-11c4-4b25-87e1-f0c4729750f9-kube-api-access-5kpvk\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.955693 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955547 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-crio-socket\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.955800 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee13f98-11c4-4b25-87e1-f0c4729750f9-tmp-dir\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.955862 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.955788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-data-volume\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.956401 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.956380 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.956737 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.956719 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee13f98-11c4-4b25-87e1-f0c4729750f9-config-volume\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.959516 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.959386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ee13f98-11c4-4b25-87e1-f0c4729750f9-metrics-tls\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.959601 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.959394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:39.959601 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.959538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e678d92f-19e7-49f2-b876-774d845ac141-cert\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.968013 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.967956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpvk\" (UniqueName: \"kubernetes.io/projected/1ee13f98-11c4-4b25-87e1-f0c4729750f9-kube-api-access-5kpvk\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:39.968100 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.968019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxg5\" (UniqueName: \"kubernetes.io/projected/e678d92f-19e7-49f2-b876-774d845ac141-kube-api-access-qjxg5\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:39.968100 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:39.968043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jhm\" (UniqueName: \"kubernetes.io/projected/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-api-access-w4jhm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:40.056096 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.056067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:02:40.098310 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.098291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:02:40.127185 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.127162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:40.347831 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.347800 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:02:40.354721 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.354700 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:02:40.354862 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.354848 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:02:40.354972 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.354958 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\"" Nov 03 16:02:40.428868 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.428838 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tc5s"] Nov 03 16:02:40.431339 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.431316 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q4t6"] Nov 03 16:02:40.432779 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:40.432736 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode678d92f_19e7_49f2_b876_774d845ac141.slice/crio-1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1 WatchSource:0}: Error finding container 1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1: Status 404 returned error can't find the container with id 1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1 Nov 03 16:02:40.435090 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:40.435060 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3725d4_d55d_42eb_896f_9d10f59cb5ef.slice/crio-9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc WatchSource:0}: Error finding container 9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc: Status 404 returned error can't find the container with id 9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc Nov 03 16:02:40.448924 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.448903 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jksqm"] Nov 03 16:02:40.460954 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:40.460934 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee13f98_11c4_4b25_87e1_f0c4729750f9.slice/crio-c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f WatchSource:0}: Error finding container c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f: Status 404 returned error can't find the container with id c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f Nov 03 16:02:40.593972 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.593941 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jksqm" event={"ID":"1ee13f98-11c4-4b25-87e1-f0c4729750f9","Type":"ContainerStarted","Data":"c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f"} Nov 03 16:02:40.595421 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.595393 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q4t6" event={"ID":"2a3725d4-d55d-42eb-896f-9d10f59cb5ef","Type":"ContainerStarted","Data":"f0c5676801c8768456ed3d4de0bd66f75717ed07f24ca42ab8d806d002bbfb4a"} Nov 03 16:02:40.595535 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.595445 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q4t6" event={"ID":"2a3725d4-d55d-42eb-896f-9d10f59cb5ef","Type":"ContainerStarted","Data":"9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc"} Nov 03 16:02:40.596515 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:40.596492 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tc5s" event={"ID":"e678d92f-19e7-49f2-b876-774d845ac141","Type":"ContainerStarted","Data":"1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1"} Nov 03 16:02:41.347388 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:41.347353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:02:41.347795 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:41.347779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:02:41.353959 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:41.353939 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\"" Nov 03 16:02:41.354576 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:41.354053 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:02:41.354576 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:41.354106 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:02:41.354576 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:41.354132 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\"" Nov 03 16:02:42.595222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.595181 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ls778"] Nov 03 16:02:42.613458 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.613417 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q4t6" event={"ID":"2a3725d4-d55d-42eb-896f-9d10f59cb5ef","Type":"ContainerStarted","Data":"ef99f4a07992f9e29094968bd872a24f6fa9e9b9c070d973502cfe10273fc85f"} Nov 03 16:02:42.613624 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.613496 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-snml5"] Nov 03 16:02:42.613692 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.613661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.627114 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.627086 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:02:42.627253 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.627155 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:02:42.627253 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.627201 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:02:42.627253 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.627090 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:02:42.627440 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.627295 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 16:02:42.627440 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.627352 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nrwgx\"" Nov 03 16:02:42.631602 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.631581 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ls778"] Nov 03 16:02:42.631739 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.631715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.676913 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.676884 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrtx4\"" Nov 03 16:02:42.677106 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.677004 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:02:42.677106 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.677035 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:02:42.724094 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.724068 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:02:42.779591 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2k6\" (UniqueName: \"kubernetes.io/projected/35065e4f-a8be-48cb-97ee-d3fb18c6117e-kube-api-access-gq2k6\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-accelerators-collector-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-textfile\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4m5\" (UniqueName: \"kubernetes.io/projected/0221604f-4dae-4b2d-8a42-76d243fa48af-kube-api-access-st4m5\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-metrics-client-ca\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-root\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-tls\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-sys\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.779814 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.780498 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-wtmp\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.780498 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.779849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0221604f-4dae-4b2d-8a42-76d243fa48af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.880638 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-sys\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.880770 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-sys\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.880770 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.880770 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-wtmp\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.880770 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0221604f-4dae-4b2d-8a42-76d243fa48af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.881009 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2k6\" (UniqueName: \"kubernetes.io/projected/35065e4f-a8be-48cb-97ee-d3fb18c6117e-kube-api-access-gq2k6\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881009 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-accelerators-collector-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881009 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-textfile\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881009 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.880922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-wtmp\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881186 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881186 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4m5\" (UniqueName: \"kubernetes.io/projected/0221604f-4dae-4b2d-8a42-76d243fa48af-kube-api-access-st4m5\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.881285 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-metrics-client-ca\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881335 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-root\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881387 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881332 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-textfile\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881387 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-tls\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881387 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.881539 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-accelerators-collector-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881591 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-root\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.881904 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881677 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0221604f-4dae-4b2d-8a42-76d243fa48af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.882015 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.881939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-metrics-client-ca\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.885703 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.885679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.887122 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.887100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-tls\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.887225 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.887153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.893634 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.893614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4m5\" (UniqueName: \"kubernetes.io/projected/0221604f-4dae-4b2d-8a42-76d243fa48af-kube-api-access-st4m5\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:42.894431 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.894410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2k6\" (UniqueName: \"kubernetes.io/projected/35065e4f-a8be-48cb-97ee-d3fb18c6117e-kube-api-access-gq2k6\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.897382 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.897352 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:42.991017 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:42.990967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:02:43.048168 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.048134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:02:43.300400 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.300367 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ls778"] Nov 03 16:02:43.304005 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:43.303960 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0221604f_4dae_4b2d_8a42_76d243fa48af.slice/crio-58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c WatchSource:0}: Error finding container 58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c: Status 404 returned error can't find the container with id 58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c Nov 03 16:02:43.609204 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.609125 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" event={"ID":"0221604f-4dae-4b2d-8a42-76d243fa48af","Type":"ContainerStarted","Data":"824d0975e41095370d5b305959d69f3f2262b7ec0e719cfd9a3db753c489f5f1"} Nov 03 16:02:43.609204 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.609179 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" event={"ID":"0221604f-4dae-4b2d-8a42-76d243fa48af","Type":"ContainerStarted","Data":"37d38b82b6d79eac5216fab05429d493a57ed65135e21507e711b022183dfeaf"} Nov 03 16:02:43.609848 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.609226 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" event={"ID":"0221604f-4dae-4b2d-8a42-76d243fa48af","Type":"ContainerStarted","Data":"58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c"} Nov 03 16:02:43.611023 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.610975 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jksqm" event={"ID":"1ee13f98-11c4-4b25-87e1-f0c4729750f9","Type":"ContainerStarted","Data":"c496608fcdde5cf53e760ba1a755fbd0acc9352ac77c982b2f4c27e2f4871ee7"} Nov 03 16:02:43.611156 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.611141 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:43.611231 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.611190 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jksqm" event={"ID":"1ee13f98-11c4-4b25-87e1-f0c4729750f9","Type":"ContainerStarted","Data":"3042d6fb002cac4cb37ce4714f64114b9c223b8f817fa31e89985f88c6ba649b"} Nov 03 16:02:43.612121 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.612100 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snml5" event={"ID":"35065e4f-a8be-48cb-97ee-d3fb18c6117e","Type":"ContainerStarted","Data":"ebfeb9552f90de45713dc5cf55906d9a92ceeabdce06c25c7a27d2d7a1703777"} Nov 03 16:02:43.632199 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.632158 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jksqm" podStartSLOduration=2.386545164 podStartE2EDuration="4.632142042s" totalImagesPullingTime="2.245596878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:43.631065341 +0000 UTC m=+39.705526126" watchObservedRunningTime="2026-11-03 16:02:43.632142042 +0000 UTC m=+39.706602830" Nov 03 16:02:43.718864 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.718839 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:02:43.733634 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.733611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.739774 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.739753 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 16:02:43.739774 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.739766 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 16:02:43.739952 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.739835 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tn6cb\"" Nov 03 16:02:43.740129 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.740116 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 16:02:43.740307 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.740294 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 16:02:43.740387 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.740376 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 16:02:43.740444 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.740422 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 16:02:43.740503 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.740455 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 16:02:43.741307 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.740940 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 16:02:43.759805 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.759771 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 16:02:43.765317 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.765285 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:02:43.889315 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889315 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889510 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889510 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889510 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889510 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889645 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889645 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889645 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889746 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889746 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889746 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.889881 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.889749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991207 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991386 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991386 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991386 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991386 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991386 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991664 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991388 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.991664 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.992741 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.992741 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.991902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.992741 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.992024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.992741 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.992099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.992741 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.992225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.992741 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.992366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.996161 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.996123 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.996919 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.996888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.997615 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.997515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.997899 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.997880 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.998154 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.998109 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.998251 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.998161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.998608 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.998588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.998667 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.998623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:43.999028 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:43.998956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:44.000926 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.000279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:44.001559 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.001537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:44.004108 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.004090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:44.150294 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.150271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:02:44.470319 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.470170 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:02:44.577751 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:44.577713 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7540b6_fcc5_4561_bc49_8ccfd4a8b616.slice/crio-8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3 WatchSource:0}: Error finding container 8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3: Status 404 returned error can't find the container with id 8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3 Nov 03 16:02:44.616894 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.616865 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3"} Nov 03 16:02:44.619122 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.619061 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tc5s" event={"ID":"e678d92f-19e7-49f2-b876-774d845ac141","Type":"ContainerStarted","Data":"48afdd3da685f27cadbddcbe3037fb9a5609d64b65e149979d33cfe7f2fbbe05"} Nov 03 16:02:44.621840 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.621814 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q4t6" event={"ID":"2a3725d4-d55d-42eb-896f-9d10f59cb5ef","Type":"ContainerStarted","Data":"1460a82e8c8693a146aca6e574918897f9c0b3ada1ad56ddae6eb260efa7925a"} Nov 03 16:02:44.642943 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.642893 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6tc5s" podStartSLOduration=2.391922312 podStartE2EDuration="5.642877044s" totalImagesPullingTime="3.250954732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:44.642327815 +0000 UTC m=+40.716788603" watchObservedRunningTime="2026-11-03 16:02:44.642877044 +0000 UTC m=+40.717337833" Nov 03 16:02:44.680918 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:44.680511 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4q4t6" podStartSLOduration=2.277078511 podStartE2EDuration="5.680488462s" totalImagesPullingTime="3.403409951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:44.679536337 +0000 UTC m=+40.753997125" watchObservedRunningTime="2026-11-03 16:02:44.680488462 +0000 UTC m=+40.754949250" Nov 03 16:02:45.161737 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.161707 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bcd5b478b-bwmwn"] Nov 03 16:02:45.178792 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.178764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.179926 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.179262 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcd5b478b-bwmwn"] Nov 03 16:02:45.185616 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.185568 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 16:02:45.185616 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.185568 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 16:02:45.185804 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.185635 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 16:02:45.187869 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.187852 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 16:02:45.187869 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.187863 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 16:02:45.188080 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.187913 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 16:02:45.188080 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.187939 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 16:02:45.188080 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.187956 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jq267\"" Nov 03 16:02:45.305298 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.305298 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.305482 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.305482 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.305482 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.305594 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.305676 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.305598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.406931 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.406893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.406931 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.406932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.407171 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.406960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.407171 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.407038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.407171 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.407068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.407171 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.407099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.407344 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.407166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.408111 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.407622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.408230 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.408132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.408230 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.408173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.408331 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.408235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.412762 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.412739 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.412849 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.412777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.420171 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.420150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.502302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.502266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:45.627485 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.627312 2578 generic.go:356] "Generic (PLEG): container finished" podID="35065e4f-a8be-48cb-97ee-d3fb18c6117e" containerID="02d31f24208070e858541426105cd7a65a3358582b2e3688cf6f65cbd289a195" exitCode=0 Nov 03 16:02:45.627890 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.627367 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snml5" event={"ID":"35065e4f-a8be-48cb-97ee-d3fb18c6117e","Type":"ContainerDied","Data":"02d31f24208070e858541426105cd7a65a3358582b2e3688cf6f65cbd289a195"} Nov 03 16:02:45.630437 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.630415 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" event={"ID":"0221604f-4dae-4b2d-8a42-76d243fa48af","Type":"ContainerStarted","Data":"6c231cc92f40a44de561580067d6a52ad79b34ea2a218d8e7006eae0b705ffbf"} Nov 03 16:02:45.679602 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.679558 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" podStartSLOduration=2.206141465 podStartE2EDuration="3.679544846s" totalImagesPullingTime="1.473403381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:45.678779025 +0000 UTC m=+41.753239838" watchObservedRunningTime="2026-11-03 16:02:45.679544846 +0000 UTC m=+41.754005633" Nov 03 16:02:45.803187 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:45.803161 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcd5b478b-bwmwn"] Nov 03 16:02:45.806154 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:45.806122 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5d7c35_6d31_483d_b2fb_13cf98fd4b6e.slice/crio-c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33 WatchSource:0}: Error finding container c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33: Status 404 returned error can't find the container with id c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33 Nov 03 16:02:46.635196 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:46.635160 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5b478b-bwmwn" event={"ID":"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e","Type":"ContainerStarted","Data":"c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33"} Nov 03 16:02:46.637626 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:46.637603 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snml5" event={"ID":"35065e4f-a8be-48cb-97ee-d3fb18c6117e","Type":"ContainerStarted","Data":"1a0104ee7c408ce05d617aff9756b72e177678ae790405cf936adcde56f6ff4a"} Nov 03 16:02:46.637747 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:46.637650 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snml5" event={"ID":"35065e4f-a8be-48cb-97ee-d3fb18c6117e","Type":"ContainerStarted","Data":"30a7e3b3905add35704fcd117014378d7690c658684aff89b04df45d80bcf6fd"} Nov 03 16:02:46.639516 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:46.639491 2578 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05" exitCode=0 Nov 03 16:02:46.639617 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:46.639583 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05"} Nov 03 16:02:46.662635 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:46.662592 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-snml5" podStartSLOduration=3.033970698 podStartE2EDuration="4.662578056s" totalImagesPullingTime="1.516547485s" totalInitContainerRuntime="112.059873ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:46.661319626 +0000 UTC m=+42.735780447" watchObservedRunningTime="2026-11-03 16:02:46.662578056 +0000 UTC m=+42.737038824" Nov 03 16:02:47.403686 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.403657 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-b64r4"] Nov 03 16:02:47.422925 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.422887 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-b64r4"] Nov 03 16:02:47.423106 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.423027 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.423264 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.423241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn24r\" (UniqueName: \"kubernetes.io/projected/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-kube-api-access-cn24r\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.423343 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.423319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.427904 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.427879 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vgckn\"" Nov 03 16:02:47.428050 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.428010 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 16:02:47.523644 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.523610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.523821 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.523667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn24r\" (UniqueName: \"kubernetes.io/projected/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-kube-api-access-cn24r\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.526869 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.526840 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.536210 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.536186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn24r\" (UniqueName: \"kubernetes.io/projected/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-kube-api-access-cn24r\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.747099 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.747060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:47.854025 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.847453 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f6b649c8c-qntdf"] Nov 03 16:02:47.891728 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.891697 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6b649c8c-qntdf"] Nov 03 16:02:47.891879 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.891836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.896581 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.896554 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 16:02:47.896700 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.896679 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 16:02:47.896774 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.896748 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 16:02:47.896855 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.896841 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 16:02:47.897960 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.897940 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 16:02:47.898201 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.898182 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r48kw\"" Nov 03 16:02:47.903336 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.903316 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 16:02:47.933776 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.933747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-federate-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.933909 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.933805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-serving-certs-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.933909 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.933835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-metrics-client-ca\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.934024 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.933918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.934024 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.933967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.934024 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.934012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.934147 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.934040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xh5\" (UniqueName: \"kubernetes.io/projected/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-kube-api-access-b9xh5\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:47.934147 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:47.934073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035436 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035436 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035648 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xh5\" (UniqueName: \"kubernetes.io/projected/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-kube-api-access-b9xh5\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035648 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035729 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-federate-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035729 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-serving-certs-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035729 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-metrics-client-ca\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.035866 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.035746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.036665 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.036342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.036665 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.036526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-serving-certs-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.036665 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.036574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-metrics-client-ca\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.038961 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.038922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-federate-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.039054 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.039037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.040597 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.040558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.040685 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.040597 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.051518 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.051496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xh5\" (UniqueName: \"kubernetes.io/projected/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-kube-api-access-b9xh5\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.073867 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.073799 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-b64r4"] Nov 03 16:02:48.076949 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:48.076921 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ae5215_0dc9_4cb1_a88d_1e637fb6ec13.slice/crio-70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8 WatchSource:0}: Error finding container 70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8: Status 404 returned error can't find the container with id 70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8 Nov 03 16:02:48.208120 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.208085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:02:48.504740 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.504713 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6b649c8c-qntdf"] Nov 03 16:02:48.511584 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:48.511557 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9523fdc9_2d03_4d32_9a63_26b5246ed5a0.slice/crio-af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5 WatchSource:0}: Error finding container af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5: Status 404 returned error can't find the container with id af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5 Nov 03 16:02:48.648888 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.648791 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" event={"ID":"9523fdc9-2d03-4d32-9a63-26b5246ed5a0","Type":"ContainerStarted","Data":"af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5"} Nov 03 16:02:48.651948 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.651926 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2"} Nov 03 16:02:48.652056 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.651977 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918"} Nov 03 16:02:48.654382 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:48.654359 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" event={"ID":"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13","Type":"ContainerStarted","Data":"70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8"} Nov 03 16:02:49.661103 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.660838 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" event={"ID":"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13","Type":"ContainerStarted","Data":"ca40e5f2b3b4284ed95b4ebe51dcd8c16d23e8e68bdd4505c7d8ccc77df8b64f"} Nov 03 16:02:49.661414 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.661160 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:49.662475 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.662443 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5b478b-bwmwn" event={"ID":"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e","Type":"ContainerStarted","Data":"fcc10e60283abe30ed74a2f360bb5441c969a83768d6e9ff2068ad74790d7f0b"} Nov 03 16:02:49.665757 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.665735 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664"} Nov 03 16:02:49.665860 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.665798 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210"} Nov 03 16:02:49.665860 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.665841 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8"} Nov 03 16:02:49.666031 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.665906 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:02:49.693356 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.693308 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" podStartSLOduration=1.226153648 podStartE2EDuration="2.69329655s" totalImagesPullingTime="1.467142902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:49.692534204 +0000 UTC m=+45.766994991" watchObservedRunningTime="2026-11-03 16:02:49.69329655 +0000 UTC m=+45.767757336" Nov 03 16:02:49.743222 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:49.743180 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bcd5b478b-bwmwn" podStartSLOduration=1.58006175 podStartE2EDuration="4.743166536s" totalImagesPullingTime="3.163104786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:49.720245496 +0000 UTC m=+45.794706283" watchObservedRunningTime="2026-11-03 16:02:49.743166536 +0000 UTC m=+45.817627323" Nov 03 16:02:50.671051 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:50.671019 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" event={"ID":"9523fdc9-2d03-4d32-9a63-26b5246ed5a0","Type":"ContainerStarted","Data":"38828e2ec91e71f825b15e4e8c08b98b067042157f33289dee86bedf620f51cb"} Nov 03 16:02:50.671424 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:50.671089 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" event={"ID":"9523fdc9-2d03-4d32-9a63-26b5246ed5a0","Type":"ContainerStarted","Data":"b3c527137885c63d07d973b03b0db15b922575e61d3a116efb90259d1a7fde7f"} Nov 03 16:02:50.671424 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:50.671109 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" event={"ID":"9523fdc9-2d03-4d32-9a63-26b5246ed5a0","Type":"ContainerStarted","Data":"47e262155bbe695a1193ef802d494a457eb2e018ca49984fbfb8e8e1452f218a"} Nov 03 16:02:50.675254 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:50.675174 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerStarted","Data":"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568"} Nov 03 16:02:50.703765 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:50.703711 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" podStartSLOduration=1.7416879650000001 podStartE2EDuration="3.703693502s" totalImagesPullingTime="1.962005537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:50.701596747 +0000 UTC m=+46.776057534" watchObservedRunningTime="2026-11-03 16:02:50.703693502 +0000 UTC m=+46.778154290" Nov 03 16:02:50.735804 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:50.735729 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.609522872 podStartE2EDuration="7.735717477s" totalImagesPullingTime="4.108881091s" totalInitContainerRuntime="17.313514ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:50.735524739 +0000 UTC m=+46.809985525" watchObservedRunningTime="2026-11-03 16:02:50.735717477 +0000 UTC m=+46.810178264" Nov 03 16:02:53.624474 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:53.624444 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jksqm" Nov 03 16:02:55.394626 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.394593 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cf46c5f77-bn7bp"] Nov 03 16:02:55.397507 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.397492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.410566 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.410543 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 16:02:55.412328 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.412307 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf46c5f77-bn7bp"] Nov 03 16:02:55.496107 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496209 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496209 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496290 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496324 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496371 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496401 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.496429 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.496419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.503187 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.503168 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:55.503283 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.503234 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:55.507836 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.507820 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:55.597776 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.597743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.597896 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.597777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.597896 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.597814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.597896 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.597874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598098 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.597945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598237 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598299 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598371 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598958 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598958 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598958 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598824 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.598958 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.598895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.599172 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.599019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.602263 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.602240 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.602263 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.602248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.611091 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.611068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.695071 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.695003 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:02:55.708084 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.708058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:02:55.988383 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:55.988362 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf46c5f77-bn7bp"] Nov 03 16:02:55.991153 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:02:55.991128 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4213e1e5_022d_4e1a_aa84_f6b0a2c13908.slice/crio-18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427 WatchSource:0}: Error finding container 18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427: Status 404 returned error can't find the container with id 18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427 Nov 03 16:02:56.706592 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:56.706554 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf46c5f77-bn7bp" event={"ID":"4213e1e5-022d-4e1a-aa84-f6b0a2c13908","Type":"ContainerStarted","Data":"8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e"} Nov 03 16:02:56.707095 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:56.706617 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf46c5f77-bn7bp" event={"ID":"4213e1e5-022d-4e1a-aa84-f6b0a2c13908","Type":"ContainerStarted","Data":"18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427"} Nov 03 16:02:56.732906 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:56.732859 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cf46c5f77-bn7bp" podStartSLOduration=1.732844204 podStartE2EDuration="1.732844204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:02:56.731431759 +0000 UTC m=+52.805892547" watchObservedRunningTime="2026-11-03 16:02:56.732844204 +0000 UTC m=+52.807304991" Nov 03 16:02:59.582026 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:02:59.582000 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:05.709026 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:05.708974 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:05.709427 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:05.709087 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:05.713463 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:05.713443 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:05.740427 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:05.740402 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:05.792464 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:05.792434 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcd5b478b-bwmwn"] Nov 03 16:03:10.118436 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.118404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:10.121765 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.121741 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:03:10.124755 ip-10-0-129-122 kubenswrapper[2578]: E1103 16:03:10.124723 2578 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:03:10.131165 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.131139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:10.190123 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.190102 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\"" Nov 03 16:03:10.197302 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.197275 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:10.219236 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.219208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:10.219369 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.219260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:10.222542 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.222521 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:03:10.222629 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.222551 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:03:10.231691 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.231663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:10.232518 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.232502 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:03:10.242046 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.242005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:10.384677 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.384601 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\"" Nov 03 16:03:10.391635 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.391590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:10.473688 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.473667 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sfkbr"] Nov 03 16:03:10.475918 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:03:10.475893 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26139b24_412b_40cd_8247_34995d62bcc9.slice/crio-9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde WatchSource:0}: Error finding container 9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde: Status 404 returned error can't find the container with id 9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde Nov 03 16:03:10.500112 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.500084 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\"" Nov 03 16:03:10.508175 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.508152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:10.663002 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.662909 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6lcf"] Nov 03 16:03:10.665960 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:03:10.665934 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216e3f72_2de8_4789_9f51_a8f13a33cd0c.slice/crio-0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3 WatchSource:0}: Error finding container 0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3: Status 404 returned error can't find the container with id 0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3 Nov 03 16:03:10.754649 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.754617 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6lcf" event={"ID":"216e3f72-2de8-4789-9f51-a8f13a33cd0c","Type":"ContainerStarted","Data":"0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3"} Nov 03 16:03:10.755633 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.755602 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfkbr" event={"ID":"26139b24-412b-40cd-8247-34995d62bcc9","Type":"ContainerStarted","Data":"9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde"} Nov 03 16:03:10.777702 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:10.777676 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5zrn"] Nov 03 16:03:10.780326 ip-10-0-129-122 kubenswrapper[2578]: W1103 16:03:10.780302 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df99de4_80b2_4cde_b4e8_aa117c1d00de.slice/crio-2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c WatchSource:0}: Error finding container 2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c: Status 404 returned error can't find the container with id 2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c Nov 03 16:03:11.761452 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:11.761367 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5zrn" event={"ID":"6df99de4-80b2-4cde-b4e8-aa117c1d00de","Type":"ContainerStarted","Data":"2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c"} Nov 03 16:03:11.763487 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:11.763461 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfkbr" event={"ID":"26139b24-412b-40cd-8247-34995d62bcc9","Type":"ContainerStarted","Data":"2b7bdbaaeb10a0317493a5073f3fd46adb8626e71afe27e005c715462ae59fc9"} Nov 03 16:03:11.763583 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:11.763513 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfkbr" event={"ID":"26139b24-412b-40cd-8247-34995d62bcc9","Type":"ContainerStarted","Data":"6a496dbf58b37a63a13684a23ea7df112c6e048921a7eac1778d719c90673a11"} Nov 03 16:03:11.785480 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:11.785402 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sfkbr" podStartSLOduration=66.739248173 podStartE2EDuration="1m7.785388112s" totalImagesPullingTime="1.046139939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:11.783902283 +0000 UTC m=+67.858363071" watchObservedRunningTime="2026-11-03 16:03:11.785388112 +0000 UTC m=+67.859848899" Nov 03 16:03:14.782138 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:14.782105 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6lcf" event={"ID":"216e3f72-2de8-4789-9f51-a8f13a33cd0c","Type":"ContainerStarted","Data":"cd2df261d8893050a1c5174a72a8bd2c3021c81f9fe65db3fbaa3815b9183527"} Nov 03 16:03:14.782593 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:14.782334 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:14.802942 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:14.802889 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n6lcf" podStartSLOduration=67.664512435 podStartE2EDuration="1m10.802877482s" totalImagesPullingTime="3.138365047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:14.802349143 +0000 UTC m=+70.876809933" watchObservedRunningTime="2026-11-03 16:03:14.802877482 +0000 UTC m=+70.877338268" Nov 03 16:03:16.413433 ip-10-0-129-122 kubenswrapper[2578]: I1103 16:03:16.413338 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:03:16.413514 ip-10-0-129-122 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 16:03:16.429372 ip-10-0-129-122 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 16:03:16.429540 ip-10-0-129-122 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 16:03:16.429710 ip-10-0-129-122 systemd[1]: kubelet.service: Consumed 4.198s CPU time. Nov 03 16:03:16.440647 ip-10-0-129-122 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:03:16.551995 ip-10-0-129-122 kubenswrapper[6319]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:03:16.551995 ip-10-0-129-122 kubenswrapper[6319]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:03:16.551995 ip-10-0-129-122 kubenswrapper[6319]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:03:16.551995 ip-10-0-129-122 kubenswrapper[6319]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:03:16.554074 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.553973 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.554074 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554074 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554079 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554082 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554086 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554089 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554092 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554094 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554097 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554100 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554102 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554105 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554107 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554110 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554112 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554115 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554117 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554120 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554122 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554125 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.554151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554128 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554130 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554132 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554135 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554137 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554140 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554142 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554145 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554153 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554156 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554160 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554163 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554165 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554168 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554170 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554173 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554175 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554178 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554180 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.554639 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554183 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554185 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554187 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554190 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554192 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554195 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554197 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554200 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554203 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554205 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554208 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554210 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554212 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554215 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554218 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554221 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554224 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554226 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554229 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554231 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.555152 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554234 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554236 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554239 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554241 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554243 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554246 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554248 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554250 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554253 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554255 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554258 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554260 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554262 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554265 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554268 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554270 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554273 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554277 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554281 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.555666 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554285 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554288 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554291 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554293 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554296 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554299 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554302 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554305 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554308 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554310 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554313 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554316 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554318 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554321 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554323 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554326 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554328 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554331 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554334 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554336 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.556220 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554338 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554341 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554343 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554346 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554350 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554354 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554357 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.554361 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555118 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555125 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555128 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555131 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555133 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555137 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555140 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555143 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555146 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555148 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555151 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555154 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.556727 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555157 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555160 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555163 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555165 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555170 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555174 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555177 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555180 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555182 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555185 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555187 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555190 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555192 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555195 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555198 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555200 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555203 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555205 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555208 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555210 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.557356 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555213 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555215 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555218 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555220 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555223 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555225 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555229 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555232 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555235 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555237 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555241 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555246 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555250 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555253 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555256 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555258 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555261 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555264 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555266 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.557864 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555269 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555271 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555274 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555276 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555279 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555281 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555284 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555286 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555289 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555291 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555293 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555296 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555298 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555301 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555303 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555306 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555309 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555311 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555314 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555316 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.558373 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555319 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555322 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555324 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555327 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555330 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555332 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555335 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555337 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555340 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555342 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555345 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555348 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555350 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555353 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555355 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555358 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555360 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555363 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555365 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.558886 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555367 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555370 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555372 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555375 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555377 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555379 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555382 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555384 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555386 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555389 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555391 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555394 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555396 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555399 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555402 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555406 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555904 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555909 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555912 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555915 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.559394 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555918 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555920 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555923 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555926 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555929 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555931 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555934 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555936 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555939 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555942 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555944 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555947 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555949 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555952 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555954 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555957 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555959 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555962 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555964 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555967 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.559887 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555970 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555972 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555975 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555977 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555996 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.555999 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556003 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556005 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556008 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556011 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556013 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556016 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556018 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556021 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556023 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556026 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556030 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556033 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556036 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.560448 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556039 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556042 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556046 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556049 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556051 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556054 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556057 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556059 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556061 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556064 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556068 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556071 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556074 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556077 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556079 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556082 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556084 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556087 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556092 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.560953 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556095 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556098 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556101 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556104 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556107 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556109 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556112 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556115 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556117 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556129 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556132 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556135 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556137 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556140 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556143 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556145 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556148 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556151 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556153 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556156 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.561471 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556159 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556161 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556164 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556166 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556169 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556171 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556174 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556176 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556179 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556181 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556184 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556188 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556190 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556193 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556196 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556199 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556202 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556204 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556206 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556209 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.561992 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556211 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556214 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556217 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.556219 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556333 6319 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556346 6319 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556356 6319 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556363 6319 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556370 6319 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556375 6319 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556380 6319 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556384 6319 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556387 6319 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556390 6319 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556394 6319 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556397 6319 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556400 6319 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556402 6319 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556405 6319 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556408 6319 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556411 6319 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556414 6319 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556419 6319 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556423 6319 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:03:16.562521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556426 6319 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556429 6319 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556432 6319 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556437 6319 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556440 6319 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556443 6319 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556446 6319 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556449 6319 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556452 6319 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556455 6319 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556458 6319 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556461 6319 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556465 6319 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556468 6319 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556472 6319 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556475 6319 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556478 6319 flags.go:64] FLAG: --enable-server="true" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556481 6319 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556485 6319 flags.go:64] FLAG: --event-burst="100" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556488 6319 flags.go:64] FLAG: --event-qps="50" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556491 6319 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556494 6319 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556497 6319 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556501 6319 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:03:16.563147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556504 6319 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556507 6319 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556511 6319 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556514 6319 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556517 6319 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556519 6319 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556522 6319 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556526 6319 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556529 6319 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556532 6319 flags.go:64] FLAG: --feature-gates="" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556536 6319 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556539 6319 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556542 6319 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556545 6319 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556548 6319 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556551 6319 flags.go:64] FLAG: --help="false" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556554 6319 flags.go:64] FLAG: --hostname-override="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556557 6319 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556560 6319 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556563 6319 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556566 6319 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556570 6319 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556576 6319 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:03:16.563801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556579 6319 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556582 6319 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556585 6319 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556588 6319 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556591 6319 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556594 6319 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556597 6319 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556600 6319 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556603 6319 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556605 6319 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556609 6319 flags.go:64] FLAG: --lock-file="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556611 6319 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556614 6319 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556617 6319 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556622 6319 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556625 6319 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556629 6319 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556632 6319 flags.go:64] FLAG: --logging-format="text" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556635 6319 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556638 6319 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556641 6319 flags.go:64] FLAG: --manifest-url="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556644 6319 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556649 6319 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556652 6319 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556656 6319 flags.go:64] FLAG: --max-pods="110" Nov 03 16:03:16.564404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556659 6319 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556662 6319 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556665 6319 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556667 6319 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556670 6319 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556673 6319 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556677 6319 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556685 6319 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556688 6319 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556691 6319 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556694 6319 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556696 6319 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556699 6319 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556702 6319 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556705 6319 flags.go:64] FLAG: --port="10250" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556708 6319 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556711 6319 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00c270b77ab1b0076" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556714 6319 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556717 6319 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556720 6319 flags.go:64] FLAG: --register-node="true" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556723 6319 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556726 6319 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556729 6319 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556733 6319 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556736 6319 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556740 6319 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:03:16.565202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556743 6319 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556746 6319 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556749 6319 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556752 6319 flags.go:64] FLAG: --runonce="false" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556755 6319 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556758 6319 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556761 6319 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556764 6319 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556767 6319 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556770 6319 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556773 6319 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556776 6319 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556780 6319 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556783 6319 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556786 6319 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556788 6319 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556791 6319 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556794 6319 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556797 6319 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556802 6319 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556806 6319 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556808 6319 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556813 6319 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556815 6319 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556818 6319 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:03:16.565864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556821 6319 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556824 6319 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556826 6319 flags.go:64] FLAG: --v="2" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556831 6319 flags.go:64] FLAG: --version="false" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556836 6319 flags.go:64] FLAG: --vmodule="" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556840 6319 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.556843 6319 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557059 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557063 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557066 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557069 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557072 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557075 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557079 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557083 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557085 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557088 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557091 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557094 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557098 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557101 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.566775 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557103 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557106 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557110 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557114 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557117 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557120 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557123 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557125 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557128 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557130 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557133 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557135 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557138 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557140 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557143 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557146 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557148 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557151 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.567585 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557154 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557156 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557158 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557161 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557164 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557166 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557169 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557172 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557174 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557176 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557179 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557181 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557185 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557187 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557190 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557192 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557195 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557197 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557200 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.568063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557202 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557204 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557207 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557210 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557212 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557215 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557217 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557220 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557222 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557225 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557227 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557229 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557232 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557234 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557237 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557239 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557242 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557244 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557247 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557249 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.568538 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557253 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557255 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557258 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557260 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557262 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557266 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557268 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557271 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557273 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557276 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557278 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557281 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557283 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557285 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557288 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557290 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557292 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557295 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557298 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557300 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.569141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557302 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557305 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557307 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557309 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557312 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557314 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557317 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557319 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557321 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557324 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557326 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557329 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557331 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557334 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557337 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.557362 6319 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:03:16.569691 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557411 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557416 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557419 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557422 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557425 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557427 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557430 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557432 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557435 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557437 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557439 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557442 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557444 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557447 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557449 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557454 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557456 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557459 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557462 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557465 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.570156 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557467 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557469 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557472 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557474 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557477 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557479 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557482 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557484 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557487 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557489 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557492 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557495 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557498 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557501 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557504 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557506 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557509 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557511 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557514 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557516 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.570644 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557518 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557521 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557523 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557526 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557529 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557531 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557533 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557537 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557540 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557542 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557546 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557549 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557553 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557555 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557558 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557561 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557564 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557567 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557570 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.571193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557572 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557576 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557579 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557582 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557585 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557588 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557592 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557594 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557597 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557599 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557602 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557604 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557607 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557609 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557612 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557614 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557617 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557619 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.571823 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557622 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557624 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557628 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557630 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557633 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557635 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557638 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557640 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557643 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557645 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557648 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557651 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557653 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557656 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557658 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557661 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557663 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557666 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557668 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557671 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.572512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557675 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557678 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557680 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557683 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557686 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557689 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557691 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557693 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.557696 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.561934 6319 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.561951 6319 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.561970 6319 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.561975 6319 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562054 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562060 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562063 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562067 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562069 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562072 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.573063 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562075 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562079 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562084 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562087 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562090 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562093 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562095 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562098 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562100 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562103 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562105 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562108 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562111 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562113 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562116 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562119 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562121 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562124 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562127 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.573546 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562129 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562132 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562135 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562137 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562146 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562149 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562151 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562154 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562156 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562159 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562161 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562164 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562166 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562169 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562171 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562174 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562176 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562178 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562181 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562183 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.574032 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562186 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562188 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562191 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562193 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562196 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562198 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562201 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562203 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562206 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562208 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562210 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562213 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562215 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562218 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562221 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562223 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562226 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562233 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562236 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562238 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.574530 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562241 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562243 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562246 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562248 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562251 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562253 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562256 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562258 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562260 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562263 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562266 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562268 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562270 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562273 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562276 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562278 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562281 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562283 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562286 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.575041 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562288 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562292 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562295 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562298 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562300 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562303 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562306 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562308 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562311 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562313 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562316 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562324 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562326 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562329 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562331 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562333 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562336 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562338 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562341 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562343 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.575501 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562346 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562348 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.562376 6319 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562561 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562565 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562568 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562571 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562573 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562576 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562579 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562581 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562584 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562586 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562588 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562591 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562594 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.576044 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562596 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562599 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562601 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562604 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562606 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562609 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562613 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562616 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562625 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562628 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562631 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562633 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562635 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562638 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562640 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562643 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562645 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562647 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.576451 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562650 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562652 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562655 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562657 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562659 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562662 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562665 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562667 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562670 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562672 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562674 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562677 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562679 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562682 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562684 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562687 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562689 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562692 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562694 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562697 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.576888 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562699 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562702 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562704 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562712 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562714 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562717 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562719 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562722 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562724 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562726 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562729 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562732 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562734 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562736 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562739 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562742 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562746 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562748 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562752 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.577398 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562755 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562757 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562760 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562762 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562765 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562767 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562770 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562772 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562775 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562777 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562780 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562782 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562784 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562787 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562789 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562791 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562794 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562802 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562805 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562807 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.577849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562809 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562812 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562814 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562817 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562819 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562822 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562824 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562827 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562829 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562831 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562834 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562837 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562839 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562841 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562844 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562846 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.578359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.562869 6319 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562905 6319 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562909 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562912 6319 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562915 6319 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562917 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562920 6319 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562922 6319 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562925 6319 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562934 6319 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562939 6319 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562942 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562946 6319 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562949 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562957 6319 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562960 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562963 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562965 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562967 6319 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:03:16.578757 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562970 6319 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562972 6319 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562975 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562977 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.562995 6319 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563000 6319 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563005 6319 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563008 6319 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563010 6319 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563013 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563015 6319 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563018 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563020 6319 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563025 6319 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563027 6319 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563030 6319 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563032 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563035 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563038 6319 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563040 6319 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:03:16.579234 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563043 6319 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563045 6319 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563048 6319 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563050 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563053 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563055 6319 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563058 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563060 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563070 6319 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563072 6319 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563075 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563078 6319 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563081 6319 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563083 6319 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563086 6319 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563088 6319 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563091 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563093 6319 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563097 6319 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563101 6319 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:03:16.579722 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563103 6319 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563106 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563108 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563111 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563113 6319 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563117 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563119 6319 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563122 6319 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563124 6319 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563127 6319 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563129 6319 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563132 6319 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563134 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563138 6319 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563141 6319 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563143 6319 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563146 6319 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563148 6319 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563151 6319 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563154 6319 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:03:16.580342 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563156 6319 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563165 6319 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563167 6319 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563170 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563172 6319 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563175 6319 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563178 6319 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563180 6319 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563182 6319 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563185 6319 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563187 6319 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563190 6319 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563192 6319 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563195 6319 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563197 6319 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563199 6319 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563202 6319 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563208 6319 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563212 6319 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563214 6319 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:03:16.580834 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563217 6319 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563219 6319 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563222 6319 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563224 6319 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563227 6319 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563231 6319 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563233 6319 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:16.563236 6319 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.563296 6319 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.564330 6319 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.564393 6319 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.564585 6319 server.go:1014] "Starting client certificate rotation" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.564734 6319 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.564804 6319 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:57:04 +0000 UTC" deadline="2028-05-31 11:58:10.515035572 +0000 UTC" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.564893 6319 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13795h54m53.950146944s" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.565170 6319 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:03:16.581414 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.567342 6319 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:03:16.581815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.569496 6319 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:03:16.581815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.571227 6319 log.go:25] "Validated CRI v1 image API" Nov 03 16:03:16.581815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.572024 6319 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:03:16.581815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.574113 6319 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a59e7546-0cd1-4458-85aa-7352cb46b66e:/dev/nvme0n1p4 fa59dc6f-986c-4a78-9e81-2e7a0c7d41b6:/dev/nvme0n1p3] Nov 03 16:03:16.581947 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.574133 6319 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c/userdata/shm major:0 minor:407 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3/userdata/shm major:0 minor:282 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde/userdata/shm major:0 minor:399 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebfeb9552f90de45713dc5cf55906d9a92ceeabdce06c25c7a27d2d7a1703777/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebfeb9552f90de45713dc5cf55906d9a92ceeabdce06c25c7a27d2d7a1703777/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~projected/kube-api-access-st4m5:{mountpoint:/var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~projected/kube-api-access-st4m5 major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0533b853-c8d6-4018-bc6c-20f99c17f1ef/volumes/kubernetes.io~projected/kube-api-access-fq64w:{mountpoint:/var/lib/kubelet/pods/0533b853-c8d6-4018-bc6c-20f99c17f1ef/volumes/kubernetes.io~projected/kube-api-access-fq64w major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ee13f98-11c4-4b25-87e1-f0c4729750f9/volumes/kubernetes.io~projected/kube-api-access-5kpvk:{mountpoint:/var/lib/kubelet/pods/1ee13f98-11c4-4b25-87e1-f0c4729750f9/volumes/kubernetes.io~projected/kube-api-access-5kpvk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ee13f98-11c4-4b25-87e1-f0c4729750f9/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1ee13f98-11c4-4b25-87e1-f0c4729750f9/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/216e3f72-2de8-4789-9f51-a8f13a33cd0c/volumes/kubernetes.io~projected/kube-api-access-bvjww:{mountpoint:/var/lib/kubelet/pods/216e3f72-2de8-4789-9f51-a8f13a33cd0c/volumes/kubernetes.io~projected/kube-api-access-bvjww major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26139b24-412b-40cd-8247-34995d62bcc9/volumes/kubernetes.io~projected/kube-api-access-47dw2:{mountpoint:/var/lib/kubelet/pods/26139b24-412b-40cd-8247-34995d62bcc9/volumes/kubernetes.io~projected/kube-api-access-47dw2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26139b24-412b-40cd-8247-34995d62bcc9/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/26139b24-412b-40cd-8247-34995d62bcc9/volumes/kubernetes.io~secret/metrics-certs major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a3725d4-d55d-42eb-896f-9d10f59cb5ef/volumes/kubernetes.io~projected/kube-api-access-w4jhm:{mountpoint:/var/lib/kubelet/pods/2a3725d4-d55d-42eb-896f-9d10f59cb5ef/volumes/kubernetes.io~projected/kube-api-access-w4jhm major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a3725d4-d55d-42eb-896f-9d10f59cb5ef/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2a3725d4-d55d-42eb-896f-9d10f59cb5ef/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30abbfa0-1328-4696-b0c7-7e5dd3947fa7/volumes/kubernetes.io~projected/kube-api-access-j9w66:{mountpoint:/var/lib/kubelet/pods/30abbfa0-1328-4696-b0c7-7e5dd3947fa7/volumes/kubernetes.io~projected/kube-api-access-j9w66 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~projected/kube-api-access-gq2k6:{mountpoint:/var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~projected/kube-api-access-gq2k6 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eab8211-6bb8-4c65-a1a1-f2d0474952a2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3eab8211-6bb8-4c65-a1a1-f2d0474952a2/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~projected/kube-api-access-9w8f9:{mountpoint:/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~projected/kube-api-access-9w8f9 major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volumes/kubernetes.io~projected/kube-api-access-pwl9d:{mountpoint:/var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volumes/kubernetes.io~projected/kube-api-access-pwl9d major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6567421d-b34c-4a99-8fad-c314c9538374/volumes/kubernetes.io~projected/kube-api-access-tf8ds:{mountpoint:/var/lib/kubelet/pods/6567421d-b34c-4a99-8fad-c314c9538374/volumes/kubernetes.io~projected/kube-api-access-tf8ds major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df99de4-80b2-4cde-b4e8-aa117c1d00de/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6df99de4-80b2-4cde-b4e8-aa117c1d00de/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7128ff3f-3a5a-40c8-98a1-2ec63572519b/volumes/kubernetes.io~projected/kube-api-access-7pmpj:{mountpoint:/var/lib/kubelet/pods/7128ff3f-3a5a-40c8-98a1-2ec63572519b/volumes/kubernetes.io~projected/kube-api-access-7pmpj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f7b62ab-f1df-47f3-99b3-b34efd73a23e/volumes/kubernetes.io~projected/kube-api-access-ttfn7:{mountpoint:/var/lib/kubelet/pods/7f7b62ab-f1df-47f3-99b3-b34efd73a23e/volumes/kubernetes.io~projected/kube-api-access-ttfn7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~projected/kube-api-access-xtlhh:{mountpoint:/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~projected/kube-api-access-xtlhh major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~projected/kube-api-access-b9xh5:{mountpoint:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~projected/kube-api-access-b9xh5 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~projected/kube-api-access-9b8ng:{mountpoint:/var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~projected/kube-api-access-9b8ng major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/alertmanager/9 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/config-reloader/6 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~empty-dir/config-out major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~projected/kube-api-access-nkqgc:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~projected/kube-api-access-nkqgc major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~projected/tls-assets major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/config-volume major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/web-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2396d77-c56c-48da-96f1-08b0287dab5f/volumes/kubernetes.io~projected/kube-api-access-h9zhv:{mountpoint:/var/lib/kubelet/pods/b2396d77-c56c-48da-96f1-08b0287dab5f/volumes/kubernetes.io~projected/kube-api-access-h9zhv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/volumes/kubernetes.io~projected/kube-api-access-cn24r:{mountpoint:/var/lib/kubelet/pods/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/volumes/kubernetes.io~projected/kube-api-access-cn24r major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e678d92f-19e7-49f2-b876-774d845ac141/volumes/kubernetes.io~projected/kube-api-access-qjxg5:{mountpoint:/var/lib/kubelet/pods/e678d92f-19e7-49f2-b876-774d845ac141/volumes/kubernetes.io~projected/kube-api-access-qjxg5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e678d92f-19e7-49f2-b876-774d845ac141/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e678d92f-19e7-49f2-b876-774d845ac141/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f105a7ecfd8aeccd987782943b526d0ac939cc8c4f1f73775b5855a92113ccdb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/19fbfda4d452e0dc386302e797e52f13a79f1525ea5dc911d24ffa8646c05eba/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/02d2bc3b1449c0eb36dcac794194a2c7ae21898d7f39182c0ed17a8765b78f55/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ff63acaf4e395ba356c4b0e0deff7bf28c21d81b54679d91efa0599c7aafd92d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cdefee6eca966c46972d72fdf9b22d27f65926b4ee36c3d12a05b29e5a4ba9a8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5d5d75fd0d6be3047dba9b071d3b274a1f4c09c0cb6f16096f321afbcf4b27ef/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a8dbeca1b6d6e1d2b135111a02ce611c1f319a2c9faedecb4f67478dd53e9a2a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/783b710c7857759935c52f0fa85dc834d3498c7872eebdcc3649058ae4771499/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3ea589657f16d8d192f3b139a9f82b83299e7f66f20337b18b11c6f0e7afc635/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/26bba13b391162572c48b04bfc2f18efff06221288145a77003bd7d679ae4a7e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/d7d550cd2d991b6511818129d0cc2814137d6dc73b40020119d10cc6a1f9392d/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/85f6288f5cf5d0deb0df170ef19a3f7187a2d69be20451161e3d471b685cd09a/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2ac6aebffc3888b77198d976dcf9e17a152a93af039ec00375e1434fb8e975f3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fcee5731a8503882dcfadbd77ce51f0c33d3299cd1766f935b0459786ed1299f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/6171cc3e0cc0bc08c4129d671fec091596e6f39b658ef4342e3cbb0a91134b9a/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cb7550c0a093d4639bd0881d7cad18d851c03f3ba7cdbd610b726e3579e87bd5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/43a1d354af09ee023d1b2f31f1645ef13d66ed99d8b8f91e845503d41e554ccf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/df06e0dc5925b614292c05792c04bf8a0473fc0bf03a216e723a40d8cd1dd3e0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a2dcdb46c7b6f7a407f2535e1af278de84c879c9a57da974b1d3984e5a919f7a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c40b6160fbddccbfe86b2f8fcdb860d5459568e78f9e68b4f78e8f79be7ecb82/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b03a37b17298462e7eb4242588acf7bdfe86c4c4932c0165d7dee487e5cc2b9d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/6f69da5b3aeb7716af86b4b36f0883d52d7af4a7eb51812451a673d1293fadd8/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/640feb6ad959c5a4283fdfc684075bc2f7503e0ea0dd5709694fc6c531a67292/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/49dda80e5ffa3e80452cdfa0c225c62758cab9e64eaceff40e8ef715de67b9be/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9aa8e07515539c93bbb97cfb4b9e664b990cacee9fd3198b8f1a2a8720d15e41/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9008014fb6a05c40062b6a9049b960cf636399bc5f4b6366591476b96bd623d8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/d67d29bbfd95e6218066bee07d35ea7877eba0ef6ac7ef161a2e20184a155793/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/370253dfdb5c23e835479bac61e6490a2fa2dbf3ba2cf56641480c57c795705f/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/5236f530aeea5e0d1efc1ddb4cca7d819e4872d553dbba244cf0615abf116642/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/596a6592f738b2c56e4f9cad056b0dc9f49203edf2bbf039527a55a9ca11d15e/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/b82825af5fbeccf81e6b97c4e6812f025dea5f2c6b75e9f85ad1e03cad3b95c2/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/d6ab5f576514209abdf60a9b0845bbe08458975a58f1652fddbfb80e8ec1916c/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/4e9a5a6a4c1e80c4228cb89bdc08e849b6e664cb0de5799be010a8444e533ff7/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/1758a14dfb2b8d91492af5f543a2fd542d7c3dc2d0be190c80c6087b110d0e81/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/ee89d577f2b57ff9d1075fa1fc7c3d1b4ac0881f45e8043777e03a2e58985808/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/4d72a9cf782844b8a1ba685eda1b6ee62c060d6bdb6ead8af63e3a0d4ee292e3/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/81e1f8a1cca9e30d893a24e1ddb78c14fa6121a3b140ae021758ef013a9228c8/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/438023831aefe80722ee95b9ee3a980c64d05307bdf3cf5b888df82c3b3b5bef/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/853590143bfc4af36b6bca9cbc63d549e4654f0c27b3b216d166341f26884969/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/5f9db2324702accba65b34928821392e26f5ee13e600864cc5ffb4aed0e8636e/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/4392b92f1caaaf5734bd3f207d8ed89945343e35cdc7210e4f5cd1f0da24b046/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/c83f329fe47c07bfc207c8cdb94c8ef4ee84a8c968a6f5a53cab6b7bb821aabd/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/a5e2a786b8eb8417db00f37aa247064084105a6f6de61df33375a6995e6d10f9/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/63bb56ca276eba982c4e6e1120a13ebaeae0a9ea21be3baba615b074138ffa9f/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/7e4b6d379fdc1aa4f8da216c7fec6cee35f77396f3abf7b5c1d56d649261987e/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/afe6ade23d1118c87d06198296069a4ce6e5ccfc92748185e85840ea1558b719/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/1b8273308968bc918a0f6a3fa71a3225f92601d0b4c633ae33933b301b7692d9/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/bdeee9b7a84939d6ba92fef2f19225dbc9c5347af4367c96ef5313e01ff41961/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/016cc756051703fd4dd2b2a4ae3f3b723b059a3aa7390cba0917b27c79a7eeac/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/02ebd4c6dd403f0701ba9c9e8f2a140ff793e3fe0761ff017c034592f16d99a1/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/541f244716bfcbae6b9d7967821bc4ac37671db414bb610afca5ac7a0933338a/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/e6b405455339a51cd14b6177aa21557a8125f2f46153bf14d22db97735e6f909/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/239df05e4eb62bf67c54312e4a26de86f83487b863a5ecad9bdc740b847242eb/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/60194b1dac0105b1fc998454af0ce669d601f190d2baf89c0325e8bb9dfd5f98/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/05e75878d33cfa92a2b5972e83d3b7229410b97cb4db62facf700778c380c590/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/20425f580d9726096c35909067e841209a9e1c9835939ac14d22479b4f575664/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/8df3873d9539bfcc20f1dae24e45bba98561add7ee3170b26fc39ea7f51855b9/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/82d40409ac896c1764aed1bac70d8c7c36333048544997bd6675e81629291690/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/f471bc1de4431774809e6dd7b497b9acbab9a071616b6c5452a21610a773aeab/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/f8042a13da646d9d2b1f592d98eaf3aef89e831a3ef3a8591c1f4b38633f65b4/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/479086a26d712c6c171749e7282621cc8e223523e8945a65ab7a88c3b90fbe24/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9b0b14a4f4b47c28f051ba1664447d4cf1fd41e867848084ee5f6b787fd00046/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/581c250b887485f011b033792890e41866ed96ff3bad862346deb21239d37a0b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/c43693713bfbd827416549d1e206ac56967e3da8f4aba199a64b95780910640c/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/7408d150e4abfaef92a6b44de47ce70e723decbc7486072b37e78a8410f2dae6/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/25433f091e8fd7824273591ac4f292a5f7eb49a18980fa816b2edb97443560a3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f47650a6c3c5971007cdc3b7956a20e0a75ac42715ee0e78f44976acbec48956/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/901dd5a6863a33698e277a19de6c8fbe1b3cd3c297ea9502a7a8774bd940f962/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ef03021f4c835d3809d0650f8c94a30d137b585e16b1ce2cddd259ec79b19d59/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/578c4db07a307e54ae3f81e96df807b54f90b4967bcec280826b6a1dbbc3bad8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8baca118960ec999296404c88a4b6691123a41a8d7c477f560fab93c76741477/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 16:03:16.582321 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.581915 6319 manager.go:217] Machine: {Timestamp:2026-11-03 16:03:16.581461065 +0000 UTC m=+0.092828924 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095038 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe9bba1ec7ecacf65149a9fc4a0e1 SystemUUID:ec2fe9bb-a1ec-7eca-cf65-149a9fc4a0e1 BootID:3500c74b-a0ad-44e1-a5e5-a760f2d8b4da Filesystems:[{Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebfeb9552f90de45713dc5cf55906d9a92ceeabdce06c25c7a27d2d7a1703777/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:382 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:317 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~projected/kube-api-access-gq2k6 DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/216e3f72-2de8-4789-9f51-a8f13a33cd0c/volumes/kubernetes.io~projected/kube-api-access-bvjww DeviceMajor:0 DeviceMinor:402 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af2f2fee0c9bb95419948696d8134adb0b3b3d6a5647cf2a64958cf68f12f30f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35065e4f-a8be-48cb-97ee-d3fb18c6117e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e678d92f-19e7-49f2-b876-774d845ac141/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9edf133e3d6c18d164165fb9fcea81eee31e7dfd55a10eb64ae11794dd7852bc/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~projected/kube-api-access-xtlhh DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/volumes/kubernetes.io~projected/kube-api-access-cn24r DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58e4c76f90744a83d8d3566d17923dfd202d2cc495b071fd6a3dbed55dee8a1c/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b2396d77-c56c-48da-96f1-08b0287dab5f/volumes/kubernetes.io~projected/kube-api-access-h9zhv DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~projected/kube-api-access-b9xh5 DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0267cd0d5c692c607659ef84ffe57ac287c616f783018b92f508f43e3edb32e3/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6567421d-b34c-4a99-8fad-c314c9538374/volumes/kubernetes.io~projected/kube-api-access-tf8ds DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd68cd48ea4169d153df2ab2caa021f0c80ee134d819c820a525e85693872a89/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e3a8c71c3ae7437a6ef9ff6bf5c3eb1934cfae989081bc40b1efd252f4fffe3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f7b62ab-f1df-47f3-99b3-b34efd73a23e/volumes/kubernetes.io~projected/kube-api-access-ttfn7 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~projected/kube-api-access-9w8f9 DeviceMajor:0 DeviceMinor:387 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26139b24-412b-40cd-8247-34995d62bcc9/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:394 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f8ab966e47294d097c3f4769df9e52229f330041c608b65303f0161389f6fde/userdata/shm DeviceMajor:0 DeviceMinor:399 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7128ff3f-3a5a-40c8-98a1-2ec63572519b/volumes/kubernetes.io~projected/kube-api-access-7pmpj DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:271 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:272 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26139b24-412b-40cd-8247-34995d62bcc9/volumes/kubernetes.io~projected/kube-api-access-47dw2 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0502bf868d9c914dda3c422ae39986d8e85805d9c854bfb44ab7f63c3a25d2f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3/userdata/shm DeviceMajor:0 DeviceMinor:282 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63f2d5ae-3734-4546-9f13-8ec3b049a83e/volumes/kubernetes.io~projected/kube-api-access-pwl9d DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0533b853-c8d6-4018-bc6c-20f99c17f1ef/volumes/kubernetes.io~projected/kube-api-access-fq64w DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c762f72e8f3cf926d1bc1857c07b48b286aac89f687c2ee7aafe31d162668d1/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0221604f-4dae-4b2d-8a42-76d243fa48af/volumes/kubernetes.io~projected/kube-api-access-st4m5 DeviceMajor:0 DeviceMinor:238 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9bb8a627352104503c0f10ea37a90ff92e42dd401b4ba58317ba450d74274a1/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ee13f98-11c4-4b25-87e1-f0c4729750f9/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:386 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5bdefb577bc84a195219a95022894c9285591dc0ecc087f05c69b6ac97661c8/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e678d92f-19e7-49f2-b876-774d845ac141/volumes/kubernetes.io~projected/kube-api-access-qjxg5 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c184456-2ccd-47b7-8b1e-5c410e3248f3/volumes/kubernetes.io~projected/kube-api-access-9b8ng DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86c66a830f9bf1a99834bd025c78851f404338dc8c515707857246792e2c7601/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes/kubernetes.io~projected/kube-api-access-nkqgc DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70bbfaef7f7227efc98a2290f8868ad4ae6c1274800b43e7a46b84293734b3c8/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9523fdc9-2d03-4d32-9a63-26b5246ed5a0/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6df99de4-80b2-4cde-b4e8-aa117c1d00de/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:401 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ee13f98-11c4-4b25-87e1-f0c4729750f9/volumes/kubernetes.io~projected/kube-api-access-5kpvk DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29a79d7e5f9bb739a49b1565e4256b9f3ab6c5c211d95acd0c12d77001338493/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a3725d4-d55d-42eb-896f-9d10f59cb5ef/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ac3c06f917b699fd8761bd4cb46165192d7d3444046aee38640d1fe9e09132c/userdata/shm DeviceMajor:0 DeviceMinor:407 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a3725d4-d55d-42eb-896f-9d10f59cb5ef/volumes/kubernetes.io~projected/kube-api-access-w4jhm DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af94af1c9b761882df44c55be86e7b37260acf44af57cb626dd45dc02ec9fad5/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3eab8211-6bb8-4c65-a1a1-f2d0474952a2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/30abbfa0-1328-4696-b0c7-7e5dd3947fa7/volumes/kubernetes.io~projected/kube-api-access-j9w66 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d147a64584f05cdeffbe5adc16e87fc89aa3d23c1b9ed59b3be25ac57f44fb61/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0267cd0d5c692c6 MacAddress:0e:b9:ca:99:55:ea Speed:10000 Mtu:8901} {Name:18ef6cad2c8881b MacAddress:82:bc:78:8e:e3:2f Speed:10000 Mtu:8901} {Name:1c762f72e8f3cf9 MacAddress:a6:98:3e:95:c0:f7 Speed:10000 Mtu:8901} {Name:2ac3c06f917b699 MacAddress:ea:84:37:b3:33:85 Speed:10000 Mtu:8901} {Name:58e4c76f90744a8 MacAddress:8e:40:06:5a:1c:98 Speed:10000 Mtu:8901} {Name:70bbfaef7f7227e MacAddress:b2:68:d1:4f:90:29 Speed:10000 Mtu:8901} {Name:8a2c63f26ed688b MacAddress:2e:a7:a5:60:ab:05 Speed:10000 Mtu:8901} {Name:9edf133e3d6c18d MacAddress:62:60:30:8d:9b:28 Speed:10000 Mtu:8901} {Name:9f8ab966e47294d MacAddress:c2:2d:ea:33:3b:9a Speed:10000 Mtu:8901} {Name:af94af1c9b76188 MacAddress:ae:a3:82:e5:d2:27 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fb:85:b9:e9 Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:10:66:1c:ef:7e Speed:0 Mtu:8901} {Name:c0502bf868d9c91 MacAddress:96:fb:ce:7b:2d:9c Speed:10000 Mtu:8901} {Name:c23245ec9657513 MacAddress:52:2d:7f:17:1f:cf Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fb:85:b9:e9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:fe:01:36:f0:66:6c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:e6:d1:4e:f5:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:03:16.582543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582326 6319 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:03:16.582543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582371 6319 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:03:16.582658 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582638 6319 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:03:16.582787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582659 6319 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-122.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:03:16.582838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582789 6319 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:03:16.582838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582796 6319 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:03:16.582838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582806 6319 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:03:16.582838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582825 6319 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:03:16.582946 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.582891 6319 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:03:16.583077 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583067 6319 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:03:16.583185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583177 6319 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:03:16.583215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583210 6319 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:03:16.583245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583220 6319 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:03:16.583245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583228 6319 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:03:16.583245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583237 6319 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:03:16.583245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583244 6319 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:03:16.583855 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583840 6319 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:03:16.583855 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.583852 6319 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:03:16.584325 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.584304 6319 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:03:16.587797 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.587780 6319 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:03:16.587797 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.587795 6319 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:03:16.587902 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.587886 6319 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 16:03:16.588106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588093 6319 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:03:16.588161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588139 6319 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:03:16.588271 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588259 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:03:16.588304 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588286 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:03:16.588304 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588298 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:03:16.588362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588311 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:03:16.588362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588323 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:03:16.588362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588333 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:03:16.588362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588342 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:03:16.588362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588352 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:03:16.588362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588362 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:03:16.588606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588371 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:03:16.588606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588384 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:03:16.588606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588402 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:03:16.588606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588438 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:03:16.588606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.588464 6319 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:03:16.591882 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.591869 6319 server.go:1263] "Started kubelet" Nov 03 16:03:16.591977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.591950 6319 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:03:16.592045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.591966 6319 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:03:16.592094 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.592048 6319 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:03:16.592388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.592370 6319 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:03:16.593086 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.593073 6319 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:03:16.593355 ip-10-0-129-122 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597161 6319 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597185 6319 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597228 6319 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:57:06 +0000 UTC" deadline="2028-08-07 08:10:20.376529182 +0000 UTC" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597264 6319 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15424h7m3.779268059s" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597490 6319 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597506 6319 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.597585 6319 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-122.us-east-2.compute.internal\" not found" Nov 03 16:03:16.598036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.597654 6319 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:03:16.600827 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.600405 6319 factory.go:55] Registering systemd factory Nov 03 16:03:16.600827 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.600429 6319 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:03:16.601001 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.600957 6319 factory.go:153] Registering CRI-O factory Nov 03 16:03:16.601001 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.601000 6319 factory.go:223] Registration of the crio container factory successfully Nov 03 16:03:16.601107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.601062 6319 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:03:16.601107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.601086 6319 factory.go:103] Registering Raw factory Nov 03 16:03:16.603951 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.601190 6319 manager.go:1199] Started watching for new ooms in manager Nov 03 16:03:16.605621 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.605603 6319 manager.go:319] Starting recovery of all containers Nov 03 16:03:16.610247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610225 6319 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:03:16.610436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610402 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63f2d5ae-3734-4546-9f13-8ec3b049a83e" volumeName="kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-config" seLinuxMountContext="" Nov 03 16:03:16.610554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610537 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e678d92f-19e7-49f2-b876-774d845ac141" volumeName="kubernetes.io/projected/e678d92f-19e7-49f2-b876-774d845ac141-kube-api-access-qjxg5" seLinuxMountContext="" Nov 03 16:03:16.610662 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610643 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26139b24-412b-40cd-8247-34995d62bcc9" volumeName="kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs" seLinuxMountContext="" Nov 03 16:03:16.610759 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610743 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3725d4-d55d-42eb-896f-9d10f59cb5ef" volumeName="kubernetes.io/empty-dir/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-data-volume" seLinuxMountContext="" Nov 03 16:03:16.610846 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610833 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35065e4f-a8be-48cb-97ee-d3fb18c6117e" volumeName="kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-metrics-client-ca" seLinuxMountContext="" Nov 03 16:03:16.610945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.610927 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63f2d5ae-3734-4546-9f13-8ec3b049a83e" volumeName="kubernetes.io/projected/63f2d5ae-3734-4546-9f13-8ec3b049a83e-kube-api-access-pwl9d" seLinuxMountContext="" Nov 03 16:03:16.611071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611055 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 16:03:16.611176 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611160 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13" volumeName="kubernetes.io/projected/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-kube-api-access-cn24r" seLinuxMountContext="" Nov 03 16:03:16.611273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611255 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ee13f98-11c4-4b25-87e1-f0c4729750f9" volumeName="kubernetes.io/empty-dir/1ee13f98-11c4-4b25-87e1-f0c4729750f9-tmp-dir" seLinuxMountContext="" Nov 03 16:03:16.611360 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611345 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9" seLinuxMountContext="" Nov 03 16:03:16.611452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611440 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63f2d5ae-3734-4546-9f13-8ec3b049a83e" volumeName="kubernetes.io/secret/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 16:03:16.611547 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611532 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 16:03:16.611635 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611619 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216e3f72-2de8-4789-9f51-a8f13a33cd0c" volumeName="kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww" seLinuxMountContext="" Nov 03 16:03:16.611712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611699 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35065e4f-a8be-48cb-97ee-d3fb18c6117e" volumeName="kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 16:03:16.611799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611784 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35065e4f-a8be-48cb-97ee-d3fb18c6117e" volumeName="kubernetes.io/empty-dir/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-textfile" seLinuxMountContext="" Nov 03 16:03:16.611896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611880 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35065e4f-a8be-48cb-97ee-d3fb18c6117e" volumeName="kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-tls" seLinuxMountContext="" Nov 03 16:03:16.611998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611967 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df99de4-80b2-4cde-b4e8-aa117c1d00de" volumeName="kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret" seLinuxMountContext="" Nov 03 16:03:16.612085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612072 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp" seLinuxMountContext="" Nov 03 16:03:16.612170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612157 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out" seLinuxMountContext="" Nov 03 16:03:16.612255 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612243 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 16:03:16.612358 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612345 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert" seLinuxMountContext="" Nov 03 16:03:16.612432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612420 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 16:03:16.612505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612493 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c184456-2ccd-47b7-8b1e-5c410e3248f3" volumeName="kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-tmp" seLinuxMountContext="" Nov 03 16:03:16.612575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612562 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:03:16.612645 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612634 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2396d77-c56c-48da-96f1-08b0287dab5f" volumeName="kubernetes.io/projected/b2396d77-c56c-48da-96f1-08b0287dab5f-kube-api-access-h9zhv" seLinuxMountContext="" Nov 03 16:03:16.612739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612703 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" volumeName="kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-binary-copy" seLinuxMountContext="" Nov 03 16:03:16.612820 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612807 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f7b62ab-f1df-47f3-99b3-b34efd73a23e" volumeName="kubernetes.io/configmap/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-serviceca" seLinuxMountContext="" Nov 03 16:03:16.612892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612879 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-metrics-client-ca" seLinuxMountContext="" Nov 03 16:03:16.612976 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.612964 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab8211-6bb8-4c65-a1a1-f2d0474952a2" volumeName="kubernetes.io/secret/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-agent-certs" seLinuxMountContext="" Nov 03 16:03:16.613069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613056 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config" seLinuxMountContext="" Nov 03 16:03:16.613164 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613151 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f7b62ab-f1df-47f3-99b3-b34efd73a23e" volumeName="kubernetes.io/projected/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-kube-api-access-ttfn7" seLinuxMountContext="" Nov 03 16:03:16.613254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613241 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:03:16.613331 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613316 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c184456-2ccd-47b7-8b1e-5c410e3248f3" volumeName="kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-tuned" seLinuxMountContext="" Nov 03 16:03:16.613449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613436 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" volumeName="kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 16:03:16.613534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613522 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63f2d5ae-3734-4546-9f13-8ec3b049a83e" volumeName="kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-script-lib" seLinuxMountContext="" Nov 03 16:03:16.613617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613604 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume" seLinuxMountContext="" Nov 03 16:03:16.613693 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611380 6319 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:03:16.613758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613687 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0533b853-c8d6-4018-bc6c-20f99c17f1ef" volumeName="kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-daemon-config" seLinuxMountContext="" Nov 03 16:03:16.613830 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613819 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26139b24-412b-40cd-8247-34995d62bcc9" volumeName="kubernetes.io/projected/26139b24-412b-40cd-8247-34995d62bcc9-kube-api-access-47dw2" seLinuxMountContext="" Nov 03 16:03:16.613908 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.613895 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert" seLinuxMountContext="" Nov 03 16:03:16.614015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614001 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6567421d-b34c-4a99-8fad-c314c9538374" volumeName="kubernetes.io/projected/6567421d-b34c-4a99-8fad-c314c9538374-kube-api-access-tf8ds" seLinuxMountContext="" Nov 03 16:03:16.614097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614084 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config" seLinuxMountContext="" Nov 03 16:03:16.614180 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614169 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/projected/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-kube-api-access-b9xh5" seLinuxMountContext="" Nov 03 16:03:16.614251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614240 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca" seLinuxMountContext="" Nov 03 16:03:16.614330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614317 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 16:03:16.614401 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614390 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3725d4-d55d-42eb-896f-9d10f59cb5ef" volumeName="kubernetes.io/projected/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-api-access-w4jhm" seLinuxMountContext="" Nov 03 16:03:16.614477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614464 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3725d4-d55d-42eb-896f-9d10f59cb5ef" volumeName="kubernetes.io/secret/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 16:03:16.614562 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614550 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config" seLinuxMountContext="" Nov 03 16:03:16.614639 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614627 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7128ff3f-3a5a-40c8-98a1-2ec63572519b" volumeName="kubernetes.io/projected/7128ff3f-3a5a-40c8-98a1-2ec63572519b-kube-api-access-7pmpj" seLinuxMountContext="" Nov 03 16:03:16.614735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614710 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert" seLinuxMountContext="" Nov 03 16:03:16.614735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614731 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614743 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614755 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.611953 6319 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614769 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7128ff3f-3a5a-40c8-98a1-2ec63572519b" volumeName="kubernetes.io/empty-dir/7128ff3f-3a5a-40c8-98a1-2ec63572519b-tmp-dir" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614784 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e678d92f-19e7-49f2-b876-774d845ac141" volumeName="kubernetes.io/secret/e678d92f-19e7-49f2-b876-774d845ac141-cert" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614799 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0221604f-4dae-4b2d-8a42-76d243fa48af" volumeName="kubernetes.io/configmap/0221604f-4dae-4b2d-8a42-76d243fa48af-metrics-client-ca" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614810 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0221604f-4dae-4b2d-8a42-76d243fa48af" volumeName="kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614833 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ee13f98-11c4-4b25-87e1-f0c4729750f9" volumeName="kubernetes.io/configmap/1ee13f98-11c4-4b25-87e1-f0c4729750f9-config-volume" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614846 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" volumeName="kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614861 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35065e4f-a8be-48cb-97ee-d3fb18c6117e" volumeName="kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614874 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca" seLinuxMountContext="" Nov 03 16:03:16.614885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614887 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614901 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614915 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0221604f-4dae-4b2d-8a42-76d243fa48af" volumeName="kubernetes.io/projected/0221604f-4dae-4b2d-8a42-76d243fa48af-kube-api-access-st4m5" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614928 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" volumeName="kubernetes.io/projected/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-kube-api-access-j9w66" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614940 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6567421d-b34c-4a99-8fad-c314c9538374" volumeName="kubernetes.io/configmap/6567421d-b34c-4a99-8fad-c314c9538374-iptables-alerter-script" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614952 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614960 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-federate-client-tls" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614968 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c184456-2ccd-47b7-8b1e-5c410e3248f3" volumeName="kubernetes.io/projected/9c184456-2ccd-47b7-8b1e-5c410e3248f3-kube-api-access-9b8ng" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.614975 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615002 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" volumeName="kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615013 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ee13f98-11c4-4b25-87e1-f0c4729750f9" volumeName="kubernetes.io/projected/1ee13f98-11c4-4b25-87e1-f0c4729750f9-kube-api-access-5kpvk" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615020 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615028 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0221604f-4dae-4b2d-8a42-76d243fa48af" volumeName="kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615035 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3725d4-d55d-42eb-896f-9d10f59cb5ef" volumeName="kubernetes.io/configmap/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615043 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35065e4f-a8be-48cb-97ee-d3fb18c6117e" volumeName="kubernetes.io/projected/35065e4f-a8be-48cb-97ee-d3fb18c6117e-kube-api-access-gq2k6" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615050 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615058 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13" volumeName="kubernetes.io/secret/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615065 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab8211-6bb8-4c65-a1a1-f2d0474952a2" volumeName="kubernetes.io/configmap/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-konnectivity-ca" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615072 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63f2d5ae-3734-4546-9f13-8ec3b049a83e" volumeName="kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-env-overrides" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615080 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-client-tls" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615087 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9523fdc9-2d03-4d32-9a63-26b5246ed5a0" volumeName="kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615094 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" volumeName="kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615102 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0533b853-c8d6-4018-bc6c-20f99c17f1ef" volumeName="kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cni-binary-copy" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615109 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0533b853-c8d6-4018-bc6c-20f99c17f1ef" volumeName="kubernetes.io/projected/0533b853-c8d6-4018-bc6c-20f99c17f1ef-kube-api-access-fq64w" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615116 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ee13f98-11c4-4b25-87e1-f0c4729750f9" volumeName="kubernetes.io/secret/1ee13f98-11c4-4b25-87e1-f0c4729750f9-metrics-tls" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615124 6319 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" volumeName="kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615132 6319 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:03:16.615644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.615137 6319 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:03:16.618856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.618832 6319 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 16:03:16.632656 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.632623 6319 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:03:16.634830 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.634810 6319 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:03:16.634830 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.634831 6319 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:03:16.634968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.634849 6319 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:03:16.634968 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.634879 6319 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 16:03:16.639218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.639197 6319 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:03:16.659769 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.659742 6319 generic.go:356] "Generic (PLEG): container finished" podID="2dd8bc170f67d79edc74165820e4f245" containerID="69ae0eb4010d371dcda55d3a0266514ef49066a70c5da055ae86f0d05fd14f6b" exitCode=0 Nov 03 16:03:16.666709 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.666642 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05" exitCode=0 Nov 03 16:03:16.676968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.676941 6319 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="7fc80880bea75f2a3a12de836909d721ccbd0d0789a3e3ccedc46142be56d843" exitCode=0 Nov 03 16:03:16.676968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.676963 6319 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="e42c5d2c0b979ad11a102797dafb1f710f4d1f9a0eb84616b1c5a27449b33990" exitCode=0 Nov 03 16:03:16.676968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.676970 6319 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="f3345b24faa1c66b2aac1cebea988d41c5b63b9ed621a533092a9ba40bf4e1e6" exitCode=0 Nov 03 16:03:16.677264 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.676975 6319 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="0b27bc782382af3e7222d0fbf8fb2d563f61485dd8a499bd15faa6428dbde655" exitCode=0 Nov 03 16:03:16.677264 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.677076 6319 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="3849d2bc5b45110716bf9d6b3878914ede1c930422fe44a0e43e89c20c4b539d" exitCode=0 Nov 03 16:03:16.677264 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.677082 6319 generic.go:356] "Generic (PLEG): container finished" podID="30abbfa0-1328-4696-b0c7-7e5dd3947fa7" containerID="e88bdbc34ae7e51ac8e013a000b5f6c64103337181d811a216997bc737d1a793" exitCode=0 Nov 03 16:03:16.687744 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.687717 6319 generic.go:356] "Generic (PLEG): container finished" podID="35065e4f-a8be-48cb-97ee-d3fb18c6117e" containerID="02d31f24208070e858541426105cd7a65a3358582b2e3688cf6f65cbd289a195" exitCode=0 Nov 03 16:03:16.707263 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.707243 6319 manager.go:324] Recovery completed Nov 03 16:03:16.729677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729650 6319 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:03:16.729677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729671 6319 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:03:16.729856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729691 6319 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:03:16.729896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729852 6319 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 16:03:16.729896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729868 6319 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 16:03:16.729896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729891 6319 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 16:03:16.730034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729898 6319 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 16:03:16.730034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729906 6319 policy_none.go:50] "Start" Nov 03 16:03:16.730034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729916 6319 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:03:16.730034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.729929 6319 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:03:16.730156 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.730082 6319 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 16:03:16.730156 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.730090 6319 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 16:03:16.730156 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.730095 6319 policy_none.go:44] "Start" Nov 03 16:03:16.734710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.734685 6319 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:03:16.735157 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.735140 6319 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 16:03:16.738255 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.738233 6319 manager.go:341] "Starting Device Plugin manager" Nov 03 16:03:16.739216 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.738258 6319 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:03:16.739216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739218 6319 server.go:87] "Starting device plugin registration server" Nov 03 16:03:16.739473 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739463 6319 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:03:16.739505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739475 6319 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:03:16.739594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739571 6319 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:03:16.739856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739680 6319 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:03:16.739856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739721 6319 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:03:16.739856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739728 6319 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:03:16.739856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.739778 6319 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:03:16.739711005Z","UUID":"d7934fbe-0e37-44c6-ba6b-b68134669f2e","Handler":null,"Name":"","Endpoint":""} Nov 03 16:03:16.741253 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.741226 6319 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:03:16.741253 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.741253 6319 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:03:16.741466 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.741445 6319 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:03:16.839975 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.839951 6319 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:03:16.841885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.841869 6319 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 16:03:16.842000 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.841897 6319 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 16:03:16.842000 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.841907 6319 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-122.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 16:03:16.842000 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.841945 6319 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.855212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.855193 6319 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.855258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.855249 6319 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.935892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.935794 6319 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal"] Nov 03 16:03:16.936476 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936435 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"2dd8bc170f67d79edc74165820e4f245","Type":"ContainerStarted","Data":"823ab6a9be1c664db9b45b01caa728e8754b2461c412d8ad9a7a6e48224af087"} Nov 03 16:03:16.936537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936498 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"2dd8bc170f67d79edc74165820e4f245","Type":"ContainerDied","Data":"69ae0eb4010d371dcda55d3a0266514ef49066a70c5da055ae86f0d05fd14f6b"} Nov 03 16:03:16.936537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936515 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"2dd8bc170f67d79edc74165820e4f245","Type":"ContainerStarted","Data":"174d2f0cb6b4811e1b39d342889b60b6b910c9973398b45ab4b0028eea16334b"} Nov 03 16:03:16.936537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936528 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"d723d695ab9475e224b694e4eda0376e","Type":"ContainerStarted","Data":"bbd64df2fd40dfcd4cea1aca1c6639b7d4f328bc8a63b908734476e4ceb1fe2d"} Nov 03 16:03:16.936667 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936539 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" event={"ID":"d723d695ab9475e224b694e4eda0376e","Type":"ContainerStarted","Data":"1f08f32bd5382b4720f23b6e37e68e67854cd0d467372c9c2e508b8cf09a5ebd"} Nov 03 16:03:16.936716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936702 6319 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.936750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.936719 6319 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.949450 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.949424 6319 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:03:16.949574 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.949495 6319 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:16.958477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:16.958456 6319 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:03:16.958545 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:16.958490 6319 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.021806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.021769 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.021806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.021807 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.022005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.021830 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d723d695ab9475e224b694e4eda0376e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"d723d695ab9475e224b694e4eda0376e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.122400 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.122369 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.122400 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.122402 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.122579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.122421 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d723d695ab9475e224b694e4eda0376e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"d723d695ab9475e224b694e4eda0376e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.122579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.122478 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d723d695ab9475e224b694e4eda0376e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"d723d695ab9475e224b694e4eda0376e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.122579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.122490 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.122579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.122493 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dd8bc170f67d79edc74165820e4f245-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" (UID: \"2dd8bc170f67d79edc74165820e4f245\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.584143 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.584094 6319 apiserver.go:52] "Watching apiserver" Nov 03 16:03:17.595562 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.595541 6319 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:03:17.596155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.596135 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xmqfv","openshift-monitoring/telemeter-client-7f6b649c8c-qntdf","openshift-dns/dns-default-jksqm","openshift-dns/node-resolver-tgtnn","openshift-monitoring/node-exporter-snml5","openshift-multus/multus-additional-cni-plugins-kjjzr","kube-system/global-pull-secret-syncer-v5zrn","openshift-monitoring/alertmanager-main-0","openshift-monitoring/monitoring-plugin-77f4f76594-b64r4","openshift-monitoring/openshift-state-metrics-7bd7489957-ls778","openshift-console/console-565565c4b5-zqhq2","openshift-insights/insights-runtime-extractor-4q4t6","openshift-network-operator/iptables-alerter-9x8lc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-lclsg","openshift-multus/multus-hpmc5","openshift-ovn-kubernetes/ovnkube-node-s7mkd","openshift-network-diagnostics/network-check-target-n6lcf","kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal","openshift-console/console-6bcd5b478b-bwmwn","openshift-console/console-cf46c5f77-bn7bp","openshift-image-registry/node-ca-n5726","openshift-ingress-canary/ingress-canary-6tc5s","openshift-multus/network-metrics-daemon-sfkbr"] Nov 03 16:03:17.603092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603066 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.603212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603123 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.603212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603141 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.603212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603161 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.603212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603176 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.603212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603199 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.603681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603236 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:03:17.603681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603240 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.603681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603303 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5rhh2\"" Nov 03 16:03:17.603681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603365 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c8h5g\"" Nov 03 16:03:17.603681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603470 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.603681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603541 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.603959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.603811 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2h492\"" Nov 03 16:03:17.604104 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604089 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r589\"" Nov 03 16:03:17.604247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604231 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.604343 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604323 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:03:17.604397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604341 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:03:17.604397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604243 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:03:17.604397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604332 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:03:17.604397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604242 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:03:17.604397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604244 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.604626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604416 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.604626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604452 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qk9pq\"" Nov 03 16:03:17.604626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604552 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dchj7\"" Nov 03 16:03:17.604626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604622 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:03:17.604864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604746 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgwbg\"" Nov 03 16:03:17.604943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.604922 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d4csl\"" Nov 03 16:03:17.605101 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.605015 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:03:17.605101 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.605027 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:03:17.605242 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.605203 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:03:17.605410 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.605393 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:03:17.605491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.605480 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.605656 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.605638 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jdvsk\"" Nov 03 16:03:17.606714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.606698 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:03:17.606786 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.606730 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.606823 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.606798 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:03:17.606909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.606888 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xkdws\"" Nov 03 16:03:17.607495 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.607476 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:03:17.607787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.607767 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c56qh\"" Nov 03 16:03:17.607787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.607770 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:03:17.607913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.607774 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.607913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.607868 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.607913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.607897 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:03:17.608114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.608012 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drdlv\"" Nov 03 16:03:17.608413 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.608383 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.608511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.608461 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:03:17.608592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.608573 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:03:17.608699 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.608684 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6bb2h\"" Nov 03 16:03:17.609022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609002 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.609136 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609111 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jk68z\"" Nov 03 16:03:17.609237 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609221 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.609307 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609288 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.609359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609349 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8szjr\"" Nov 03 16:03:17.609455 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609440 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:03:17.609701 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.609684 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:03:17.610018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.610004 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:03:17.610106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.610044 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nrwgx\"" Nov 03 16:03:17.612558 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.612529 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.614239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.614223 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:03:17.614975 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.614956 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.615123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615093 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.615228 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615146 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:03:17.615228 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615179 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 16:03:17.615388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615369 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 16:03:17.615457 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615417 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 16:03:17.615457 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615421 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 16:03:17.615559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615472 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 16:03:17.615559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615494 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:03:17.615559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615500 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrtx4\"" Nov 03 16:03:17.615559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615503 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 16:03:17.615758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615745 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 16:03:17.615758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615750 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 16:03:17.615866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.615798 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 16:03:17.616180 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.616167 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:03:17.617038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.617023 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tn6cb\"" Nov 03 16:03:17.618784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.618766 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 16:03:17.624383 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624366 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zhv\" (UniqueName: \"kubernetes.io/projected/b2396d77-c56c-48da-96f1-08b0287dab5f-kube-api-access-h9zhv\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.624463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624390 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:17.624463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624407 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-script-lib\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.624463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624424 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6567421d-b34c-4a99-8fad-c314c9538374-iptables-alerter-script\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.624623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624559 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-socket-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.624623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624603 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7128ff3f-3a5a-40c8-98a1-2ec63572519b-tmp-dir\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.624720 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624632 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-script-lib\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.624720 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624673 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmpj\" (UniqueName: \"kubernetes.io/projected/7128ff3f-3a5a-40c8-98a1-2ec63572519b-kube-api-access-7pmpj\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.624815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624714 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.624815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624752 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9w66\" (UniqueName: \"kubernetes.io/projected/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-kube-api-access-j9w66\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.624815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.624757 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7128ff3f-3a5a-40c8-98a1-2ec63572519b-tmp-dir\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.625178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625153 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-node-log\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625208 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-netd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625252 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-config\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625281 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6567421d-b34c-4a99-8fad-c314c9538374-iptables-alerter-script\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.625395 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625309 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-env-overrides\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625442 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625408 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7128ff3f-3a5a-40c8-98a1-2ec63572519b-hosts-file\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.625571 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625546 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovnkube-config\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625576 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-slash\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625584 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63f2d5ae-3734-4546-9f13-8ec3b049a83e-env-overrides\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625609 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625684 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovn-node-metrics-cert\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625736 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cnibin\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625767 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625814 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-device-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625842 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-sys-fs\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625874 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-systemd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625919 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-var-lib-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.625970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.625951 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626003 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.626392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626041 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.626392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626085 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-systemd-units\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626262 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.626392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626307 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.626392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626354 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6567421d-b34c-4a99-8fad-c314c9538374-host-slash\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.626726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626400 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.626726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626434 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-bin\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626472 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwl9d\" (UniqueName: \"kubernetes.io/projected/63f2d5ae-3734-4546-9f13-8ec3b049a83e-kube-api-access-pwl9d\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626527 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63f2d5ae-3734-4546-9f13-8ec3b049a83e-ovn-node-metrics-cert\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626700 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-kubelet\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626753 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-netns\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626813 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-ovn\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626843 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-log-socket\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626877 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-etc-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.626950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.626920 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.627196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.627018 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.627196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.627056 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8ds\" (UniqueName: \"kubernetes.io/projected/6567421d-b34c-4a99-8fad-c314c9538374-kube-api-access-tf8ds\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.627256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.627246 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.627324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.627304 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-registration-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.627388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.627346 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.635648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.635623 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmpj\" (UniqueName: \"kubernetes.io/projected/7128ff3f-3a5a-40c8-98a1-2ec63572519b-kube-api-access-7pmpj\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.635732 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.635712 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zhv\" (UniqueName: \"kubernetes.io/projected/b2396d77-c56c-48da-96f1-08b0287dab5f-kube-api-access-h9zhv\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.635850 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.635834 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjww\" (UniqueName: \"kubernetes.io/projected/216e3f72-2de8-4789-9f51-a8f13a33cd0c-kube-api-access-bvjww\") pod \"network-check-target-n6lcf\" (UID: \"216e3f72-2de8-4789-9f51-a8f13a33cd0c\") " pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:17.636632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.636611 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9w66\" (UniqueName: \"kubernetes.io/projected/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-kube-api-access-j9w66\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.639024 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.639005 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwl9d\" (UniqueName: \"kubernetes.io/projected/63f2d5ae-3734-4546-9f13-8ec3b049a83e-kube-api-access-pwl9d\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.640075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.640060 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8ds\" (UniqueName: \"kubernetes.io/projected/6567421d-b34c-4a99-8fad-c314c9538374-kube-api-access-tf8ds\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.695179 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.695156 6319 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.695298 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.695266 6319 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.699621 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.699602 6319 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:03:17.701090 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701063 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 16:03:17.701192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701175 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 16:03:17.701258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701193 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 16:03:17.701619 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701580 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 16:03:17.701619 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701605 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701628 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vgckn\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701613 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r48kw\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701681 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701709 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701718 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jq267\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701721 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701720 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 16:03:17.701778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701743 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 16:03:17.702155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701794 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 16:03:17.702155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701896 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 16:03:17.702155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.701926 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 16:03:17.707113 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.707094 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 16:03:17.707188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.707155 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 16:03:17.707538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.707525 6319 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:03:17.707573 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:17.707564 6319 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.708252 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.708239 6319 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:03:17.708299 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:17.708269 6319 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-122.us-east-2.compute.internal" Nov 03 16:03:17.727853 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727828 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.727957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727854 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-etc-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.727957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727874 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpvk\" (UniqueName: \"kubernetes.io/projected/1ee13f98-11c4-4b25-87e1-f0c4729750f9-kube-api-access-5kpvk\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.727957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727891 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xh5\" (UniqueName: \"kubernetes.io/projected/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-kube-api-access-b9xh5\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.727957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727913 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-serving-certs-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.727957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727937 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727954 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-etc-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.727964 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8b2287e-d813-4fb2-bea3-5443013af667-tmp\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728013 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qchgd\" (UniqueName: \"kubernetes.io/projected/c8b2287e-d813-4fb2-bea3-5443013af667-kube-api-access-qchgd\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728089 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728097 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-run\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728138 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-tuned\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728164 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728187 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-serving-certs-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728192 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-node-log\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.728216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728192 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728236 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-tuned\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728240 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-node-log\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728306 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cnibin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728328 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728338 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728344 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728370 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728404 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-socket-dir-parent\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728503 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728517 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728535 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysconfig\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.728559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728550 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-tmp\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728568 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4m5\" (UniqueName: \"kubernetes.io/projected/0221604f-4dae-4b2d-8a42-76d243fa48af-kube-api-access-st4m5\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728577 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728616 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-bin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728642 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728660 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c184456-2ccd-47b7-8b1e-5c410e3248f3-tmp\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728675 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728766 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728776 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728798 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-crio-socket\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728825 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728860 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728848 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-kubelet-config\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728919 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728939 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.728917 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729000 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0221604f-4dae-4b2d-8a42-76d243fa48af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729030 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729054 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee13f98-11c4-4b25-87e1-f0c4729750f9-tmp-dir\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729075 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df99de4-80b2-4cde-b4e8-aa117c1d00de-original-pull-secret\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729079 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.729089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729082 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729125 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729141 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729143 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729146 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee13f98-11c4-4b25-87e1-f0c4729750f9-tmp-dir\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729175 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729162 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-kubelet\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729213 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-log-socket\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729175 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0221604f-4dae-4b2d-8a42-76d243fa48af-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729265 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-kubelet\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729275 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729304 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729304 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-log-socket\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729303 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729332 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729343 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729380 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-trusted-ca-bundle\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729409 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-tls\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729433 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-metrics-client-ca\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729457 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729468 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729472 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-conf-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729486 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729508 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729512 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729581 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-lib-modules\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729612 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8ng\" (UniqueName: \"kubernetes.io/projected/9c184456-2ccd-47b7-8b1e-5c410e3248f3-kube-api-access-9b8ng\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729621 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729627 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxg5\" (UniqueName: \"kubernetes.io/projected/e678d92f-19e7-49f2-b876-774d845ac141-kube-api-access-qjxg5\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729645 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-textfile\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729646 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-metrics-client-ca\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729655 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-tls\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729659 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-netd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729697 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-netd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729716 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-textfile\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729716 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-multus-certs\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729764 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jhm\" (UniqueName: \"kubernetes.io/projected/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-api-access-w4jhm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729793 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-sys\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729820 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729843 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-slash\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729869 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729896 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729920 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-device-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729928 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-slash\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729946 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729973 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-metrics-client-ca\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729977 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-device-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.729927 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730019 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730033 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.730026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730051 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-kubernetes\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730066 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730083 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730129 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730161 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-multus\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730179 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-metrics-client-ca\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730164 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730201 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn24r\" (UniqueName: \"kubernetes.io/projected/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-kube-api-access-cn24r\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730223 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730228 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-netns\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730256 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-ovn\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730276 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-netns\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730284 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730313 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-kubelet\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730328 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-ovn\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730346 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730380 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2k6\" (UniqueName: \"kubernetes.io/projected/35065e4f-a8be-48cb-97ee-d3fb18c6117e-kube-api-access-gq2k6\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730387 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730382 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730407 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-registration-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730451 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-netns\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730472 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-daemon-config\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730491 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730500 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee13f98-11c4-4b25-87e1-f0c4729750f9-config-volume\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730505 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-registration-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730540 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-dbus\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730550 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0221604f-4dae-4b2d-8a42-76d243fa48af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730661 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-agent-certs\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730708 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee13f98-11c4-4b25-87e1-f0c4729750f9-config-volume\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730838 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-daemon-config\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730876 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cni-binary-copy\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730910 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-host\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730912 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-agent-certs\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.730934 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-federate-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731075 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cni-binary-copy\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731105 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e678d92f-19e7-49f2-b876-774d845ac141-cert\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731130 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-k8s-cni-cncf-io\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731139 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-federate-client-tls\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731147 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-modprobe-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731186 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-accelerators-collector-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731211 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7128ff3f-3a5a-40c8-98a1-2ec63572519b-hosts-file\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731233 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731273 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731294 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731317 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731335 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731339 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731503 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7128ff3f-3a5a-40c8-98a1-2ec63572519b-hosts-file\") pod \"node-resolver-tgtnn\" (UID: \"7128ff3f-3a5a-40c8-98a1-2ec63572519b\") " pod="openshift-dns/node-resolver-tgtnn" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731520 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ee13f98-11c4-4b25-87e1-f0c4729750f9-metrics-tls\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731592 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731650 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfn7\" (UniqueName: \"kubernetes.io/projected/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-kube-api-access-ttfn7\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731651 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731676 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731684 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e678d92f-19e7-49f2-b876-774d845ac141-cert\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731723 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-accelerators-collector-config\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731804 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-service-ca\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731856 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6567421d-b34c-4a99-8fad-c314c9538374-host-slash\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731902 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6567421d-b34c-4a99-8fad-c314c9538374-host-slash\") pod \"iptables-alerter-9x8lc\" (UID: \"6567421d-b34c-4a99-8fad-c314c9538374\") " pod="openshift-network-operator/iptables-alerter-9x8lc" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731924 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731940 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-system-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.731971 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732014 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b2287e-d813-4fb2-bea3-5443013af667-console-serving-cert\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732054 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.732053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732075 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732117 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732143 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732159 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732193 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-data-volume\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732322 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ee13f98-11c4-4b25-87e1-f0c4729750f9-metrics-tls\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732337 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732371 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732400 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-data-volume\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732448 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732572 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-console-config\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732614 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732689 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-socket-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732705 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732690 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732740 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-socket-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732772 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-wtmp\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732827 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26139b24-412b-40cd-8247-34995d62bcc9-metrics-certs\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732836 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.732887 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq64w\" (UniqueName: \"kubernetes.io/projected/0533b853-c8d6-4018-bc6c-20f99c17f1ef-kube-api-access-fq64w\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733023 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733106 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cnibin\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733207 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-hostroot\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733220 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30abbfa0-1328-4696-b0c7-7e5dd3947fa7-cnibin\") pod \"multus-additional-cni-plugins-kjjzr\" (UID: \"30abbfa0-1328-4696-b0c7-7e5dd3947fa7\") " pod="openshift-multus/multus-additional-cni-plugins-kjjzr" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733234 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733268 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-etc-kubernetes\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733309 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-konnectivity-ca\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733352 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-serviceca\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733406 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-oauth-serving-cert\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733431 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-sys-fs\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733480 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-systemd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733504 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-var-lib-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733529 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733528 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3eab8211-6bb8-4c65-a1a1-f2d0474952a2-konnectivity-ca\") pod \"konnectivity-agent-xmqfv\" (UID: \"3eab8211-6bb8-4c65-a1a1-f2d0474952a2\") " pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733591 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-run-systemd\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733631 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-var-lib-openvswitch\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733650 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-sys-fs\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733681 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733766 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733798 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733839 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b2287e-d813-4fb2-bea3-5443013af667-console-oauth-config\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733865 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-systemd-units\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733888 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-conf\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733920 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-systemd\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733945 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-host\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.733972 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734022 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-bin\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734047 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734071 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-var-lib-kubelet\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734168 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-secret-telemeter-client\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734179 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-serviceca\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734223 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2396d77-c56c-48da-96f1-08b0287dab5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5pzn\" (UID: \"b2396d77-c56c-48da-96f1-08b0287dab5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5pzn" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734227 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-host-cni-bin\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734249 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63f2d5ae-3734-4546-9f13-8ec3b049a83e-systemd-units\") pod \"ovnkube-node-s7mkd\" (UID: \"63f2d5ae-3734-4546-9f13-8ec3b049a83e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734164 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-root\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734362 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-sys\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734405 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dw2\" (UniqueName: \"kubernetes.io/projected/26139b24-412b-40cd-8247-34995d62bcc9-kube-api-access-47dw2\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:17.734791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.734454 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.741871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.739298 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xh5\" (UniqueName: \"kubernetes.io/projected/9523fdc9-2d03-4d32-9a63-26b5246ed5a0-kube-api-access-b9xh5\") pod \"telemeter-client-7f6b649c8c-qntdf\" (UID: \"9523fdc9-2d03-4d32-9a63-26b5246ed5a0\") " pod="openshift-monitoring/telemeter-client-7f6b649c8c-qntdf" Nov 03 16:03:17.741871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.739471 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4m5\" (UniqueName: \"kubernetes.io/projected/0221604f-4dae-4b2d-8a42-76d243fa48af-kube-api-access-st4m5\") pod \"openshift-state-metrics-7bd7489957-ls778\" (UID: \"0221604f-4dae-4b2d-8a42-76d243fa48af\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ls778" Nov 03 16:03:17.741871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.739836 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpvk\" (UniqueName: \"kubernetes.io/projected/1ee13f98-11c4-4b25-87e1-f0c4729750f9-kube-api-access-5kpvk\") pod \"dns-default-jksqm\" (UID: \"1ee13f98-11c4-4b25-87e1-f0c4729750f9\") " pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:17.741871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.739875 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8ng\" (UniqueName: \"kubernetes.io/projected/9c184456-2ccd-47b7-8b1e-5c410e3248f3-kube-api-access-9b8ng\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.741871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.740406 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jhm\" (UniqueName: \"kubernetes.io/projected/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-kube-api-access-w4jhm\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.741871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.740767 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") pod \"console-6bcd5b478b-bwmwn\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:17.743600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.743563 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq64w\" (UniqueName: \"kubernetes.io/projected/0533b853-c8d6-4018-bc6c-20f99c17f1ef-kube-api-access-fq64w\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.743600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.743565 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxg5\" (UniqueName: \"kubernetes.io/projected/e678d92f-19e7-49f2-b876-774d845ac141-kube-api-access-qjxg5\") pod \"ingress-canary-6tc5s\" (UID: \"e678d92f-19e7-49f2-b876-774d845ac141\") " pod="openshift-ingress-canary/ingress-canary-6tc5s" Nov 03 16:03:17.743773 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.743591 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfn7\" (UniqueName: \"kubernetes.io/projected/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-kube-api-access-ttfn7\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.743913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.743892 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") pod \"alertmanager-main-0\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:17.744164 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.744142 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2k6\" (UniqueName: \"kubernetes.io/projected/35065e4f-a8be-48cb-97ee-d3fb18c6117e-kube-api-access-gq2k6\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.744289 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.744269 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn24r\" (UniqueName: \"kubernetes.io/projected/d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13-kube-api-access-cn24r\") pod \"monitoring-plugin-77f4f76594-b64r4\" (UID: \"d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:03:17.744338 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.744294 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") pod \"console-cf46c5f77-bn7bp\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:17.744743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.744730 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dw2\" (UniqueName: \"kubernetes.io/projected/26139b24-412b-40cd-8247-34995d62bcc9-kube-api-access-47dw2\") pod \"network-metrics-daemon-sfkbr\" (UID: \"26139b24-412b-40cd-8247-34995d62bcc9\") " pod="openshift-multus/network-metrics-daemon-sfkbr" Nov 03 16:03:17.835649 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835573 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-wtmp\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.835649 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835611 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-hostroot\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.835649 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835628 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-etc-kubernetes\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835649 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-oauth-serving-cert\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835671 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835686 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b2287e-d813-4fb2-bea3-5443013af667-console-oauth-config\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835692 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-etc-kubernetes\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835692 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-hostroot\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835702 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-conf\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835752 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835764 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-systemd\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835791 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-host\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835725 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-node-exporter-wtmp\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835806 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-systemd\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835794 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-conf\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835817 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-var-lib-kubelet\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835842 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-root\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835857 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-sys\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835859 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-var-lib-kubelet\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835861 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-host\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.835887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835881 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8b2287e-d813-4fb2-bea3-5443013af667-tmp\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835911 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-root\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835938 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qchgd\" (UniqueName: \"kubernetes.io/projected/c8b2287e-d813-4fb2-bea3-5443013af667-kube-api-access-qchgd\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.835953 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-run\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836023 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cnibin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836062 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-socket-dir-parent\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836089 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysconfig\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836106 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-run\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836116 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-bin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836152 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-crio-socket\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836179 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-kubelet-config\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836176 6319 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836196 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-socket-dir-parent\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836205 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836154 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-cnibin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836234 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-trusted-ca-bundle\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836237 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysconfig\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836242 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a3725d4-d55d-42eb-896f-9d10f59cb5ef-crio-socket\") pod \"insights-runtime-extractor-4q4t6\" (UID: \"2a3725d4-d55d-42eb-896f-9d10f59cb5ef\") " pod="openshift-insights/insights-runtime-extractor-4q4t6" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836152 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35065e4f-a8be-48cb-97ee-d3fb18c6117e-sys\") pod \"node-exporter-snml5\" (UID: \"35065e4f-a8be-48cb-97ee-d3fb18c6117e\") " pod="openshift-monitoring/node-exporter-snml5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836259 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8b2287e-d813-4fb2-bea3-5443013af667-tmp\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836290 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-kubelet-config\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836328 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-conf-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836187 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-bin\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836359 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-lib-modules\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836369 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-sysctl-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836384 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-multus-conf-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836402 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-multus-certs\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836437 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-multus-certs\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836462 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-sys\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836500 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-sys\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836509 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-oauth-serving-cert\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836475 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-lib-modules\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836537 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-kubernetes\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836554 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-multus\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836573 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-kubelet\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836577 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-kubernetes\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836598 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-netns\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836617 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-kubelet\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836620 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-var-lib-cni-multus\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836621 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-dbus\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836646 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-netns\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836660 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-host\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836671 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df99de4-80b2-4cde-b4e8-aa117c1d00de-dbus\") pod \"global-pull-secret-syncer-v5zrn\" (UID: \"6df99de4-80b2-4cde-b4e8-aa117c1d00de\") " pod="kube-system/global-pull-secret-syncer-v5zrn" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836687 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-k8s-cni-cncf-io\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836709 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-modprobe-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836712 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f7b62ab-f1df-47f3-99b3-b34efd73a23e-host\") pod \"node-ca-n5726\" (UID: \"7f7b62ab-f1df-47f3-99b3-b34efd73a23e\") " pod="openshift-image-registry/node-ca-n5726" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836737 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-host-run-k8s-cni-cncf-io\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836730 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-service-ca\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836767 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-system-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836793 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b2287e-d813-4fb2-bea3-5443013af667-console-serving-cert\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836820 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-console-config\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836830 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c184456-2ccd-47b7-8b1e-5c410e3248f3-etc-modprobe-d\") pod \"tuned-lclsg\" (UID: \"9c184456-2ccd-47b7-8b1e-5c410e3248f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lclsg" Nov 03 16:03:17.836884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.836882 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0533b853-c8d6-4018-bc6c-20f99c17f1ef-system-cni-dir\") pod \"multus-hpmc5\" (UID: \"0533b853-c8d6-4018-bc6c-20f99c17f1ef\") " pod="openshift-multus/multus-hpmc5" Nov 03 16:03:17.839015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.837137 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-trusted-ca-bundle\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.839015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.837230 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-service-ca\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.839015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.837368 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b2287e-d813-4fb2-bea3-5443013af667-console-config\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.839015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.838812 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b2287e-d813-4fb2-bea3-5443013af667-console-oauth-config\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.839241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.839220 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b2287e-d813-4fb2-bea3-5443013af667-console-serving-cert\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.848706 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.848679 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qchgd\" (UniqueName: \"kubernetes.io/projected/c8b2287e-d813-4fb2-bea3-5443013af667-kube-api-access-qchgd\") pod \"console-565565c4b5-zqhq2\" (UID: \"c8b2287e-d813-4fb2-bea3-5443013af667\") " pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:17.923514 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:17.923483 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:18.196303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.196234 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565565c4b5-zqhq2"] Nov 03 16:03:18.199640 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:18.199612 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b2287e_d813_4fb2_bea3_5443013af667.slice/crio-c0b957cf7f9b2c7213380d142f72e33ba5dcd6c41db36b6121228ef9d70ee63f WatchSource:0}: Error finding container c0b957cf7f9b2c7213380d142f72e33ba5dcd6c41db36b6121228ef9d70ee63f: Status 404 returned error can't find the container with id c0b957cf7f9b2c7213380d142f72e33ba5dcd6c41db36b6121228ef9d70ee63f Nov 03 16:03:18.541248 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.541219 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:03:18.699411 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.699376 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565565c4b5-zqhq2" event={"ID":"c8b2287e-d813-4fb2-bea3-5443013af667","Type":"ContainerStarted","Data":"ec79c9ac92682c72eb3ff0d603675e4c633a8c040c493104f9f4e6d499b18fde"} Nov 03 16:03:18.699825 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.699426 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565565c4b5-zqhq2" event={"ID":"c8b2287e-d813-4fb2-bea3-5443013af667","Type":"ContainerStarted","Data":"c0b957cf7f9b2c7213380d142f72e33ba5dcd6c41db36b6121228ef9d70ee63f"} Nov 03 16:03:18.700154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.699944 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="alertmanager" containerID="cri-o://00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918" gracePeriod=120 Nov 03 16:03:18.700154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.700019 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="config-reloader" containerID="cri-o://ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2" gracePeriod=120 Nov 03 16:03:18.700154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.700020 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-web" containerID="cri-o://932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8" gracePeriod=120 Nov 03 16:03:18.700154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.699970 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-metric" containerID="cri-o://761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664" gracePeriod=120 Nov 03 16:03:18.700154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.699948 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="prom-label-proxy" containerID="cri-o://e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568" gracePeriod=120 Nov 03 16:03:18.700154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.699969 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy" containerID="cri-o://62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210" gracePeriod=120 Nov 03 16:03:18.722624 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.722584 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-565565c4b5-zqhq2" podStartSLOduration=2.722572075 podStartE2EDuration="2.722572075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:18.722331315 +0000 UTC m=+2.233699184" watchObservedRunningTime="2026-11-03 16:03:18.722572075 +0000 UTC m=+2.233939933" Nov 03 16:03:18.996226 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:18.996195 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jksqm" Nov 03 16:03:19.706037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706001 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568" exitCode=0 Nov 03 16:03:19.706037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706037 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210" exitCode=0 Nov 03 16:03:19.706432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706045 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2" exitCode=0 Nov 03 16:03:19.706432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706051 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918" exitCode=0 Nov 03 16:03:19.706432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706014 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568"} Nov 03 16:03:19.706432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706167 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210"} Nov 03 16:03:19.706432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706197 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2"} Nov 03 16:03:19.706432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.706226 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918"} Nov 03 16:03:19.937879 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:19.937855 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.056060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056030 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056072 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056104 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056127 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056165 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056181 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056202 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056275 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056301 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056334 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056369 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056401 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056438 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") pod \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\" (UID: \"af7540b6-fcc5-4561-bc49-8ccfd4a8b616\") " Nov 03 16:03:20.056576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056555 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:20.056905 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.056706 6319 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.057106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.057086 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:20.059182 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.059154 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:20.059715 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.059687 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.059854 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.059834 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "kube-api-access-nkqgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:20.059932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.059915 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.060740 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.060707 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.061090 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.061060 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.061256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.061230 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.061344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.061321 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:20.061383 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.061350 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:20.064427 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.064345 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.069650 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.069626 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config" pod "af7540b6-fcc5-4561-bc49-8ccfd4a8b616" (UID: "af7540b6-fcc5-4561-bc49-8ccfd4a8b616"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:20.157253 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157228 6319 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-out\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157253 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157252 6319 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157264 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqgc\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-kube-api-access-nkqgc\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157275 6319 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-tls-assets\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157283 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-metrics-client-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157292 6319 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157301 6319 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-main-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157310 6319 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157319 6319 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157326 6319 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-alertmanager-main-db\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157335 6319 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-cluster-tls-config\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.157366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.157343 6319 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af7540b6-fcc5-4561-bc49-8ccfd4a8b616-web-config\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:20.713197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713169 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664" exitCode=0 Nov 03 16:03:20.713197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713192 6319 generic.go:356] "Generic (PLEG): container finished" podID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerID="932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8" exitCode=0 Nov 03 16:03:20.713616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713250 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.713616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713261 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664"} Nov 03 16:03:20.713616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713333 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8"} Nov 03 16:03:20.713616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713364 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af7540b6-fcc5-4561-bc49-8ccfd4a8b616","Type":"ContainerDied","Data":"8a2c63f26ed688bfe5aa4fad151c2c7211bcfba2e7500dd65df57354a7b707b3"} Nov 03 16:03:20.713616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.713413 6319 scope.go:122] "RemoveContainer" containerID="e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568" Nov 03 16:03:20.720974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.720950 6319 scope.go:122] "RemoveContainer" containerID="761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664" Nov 03 16:03:20.727709 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.727692 6319 scope.go:122] "RemoveContainer" containerID="62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210" Nov 03 16:03:20.734143 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.734118 6319 scope.go:122] "RemoveContainer" containerID="932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8" Nov 03 16:03:20.736534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.736509 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:03:20.741130 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.741110 6319 scope.go:122] "RemoveContainer" containerID="ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2" Nov 03 16:03:20.741299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.741282 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:03:20.748113 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.748097 6319 scope.go:122] "RemoveContainer" containerID="00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918" Nov 03 16:03:20.754572 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.754559 6319 scope.go:122] "RemoveContainer" containerID="1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05" Nov 03 16:03:20.761046 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761031 6319 scope.go:122] "RemoveContainer" containerID="e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568" Nov 03 16:03:20.761300 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.761280 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568\": container with ID starting with e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568 not found: ID does not exist" containerID="e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568" Nov 03 16:03:20.761349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761310 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568"} err="failed to get container status \"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568\": rpc error: code = NotFound desc = could not find container \"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568\": container with ID starting with e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568 not found: ID does not exist" Nov 03 16:03:20.761349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761340 6319 scope.go:122] "RemoveContainer" containerID="761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664" Nov 03 16:03:20.761550 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.761530 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664\": container with ID starting with 761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664 not found: ID does not exist" containerID="761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664" Nov 03 16:03:20.761597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761557 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664"} err="failed to get container status \"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664\": rpc error: code = NotFound desc = could not find container \"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664\": container with ID starting with 761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664 not found: ID does not exist" Nov 03 16:03:20.761597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761572 6319 scope.go:122] "RemoveContainer" containerID="62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210" Nov 03 16:03:20.761827 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.761808 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210\": container with ID starting with 62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210 not found: ID does not exist" containerID="62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210" Nov 03 16:03:20.761899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761835 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210"} err="failed to get container status \"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210\": rpc error: code = NotFound desc = could not find container \"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210\": container with ID starting with 62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210 not found: ID does not exist" Nov 03 16:03:20.761899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.761854 6319 scope.go:122] "RemoveContainer" containerID="932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8" Nov 03 16:03:20.762112 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.762093 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8\": container with ID starting with 932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8 not found: ID does not exist" containerID="932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8" Nov 03 16:03:20.762148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762117 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8"} err="failed to get container status \"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8\": rpc error: code = NotFound desc = could not find container \"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8\": container with ID starting with 932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8 not found: ID does not exist" Nov 03 16:03:20.762148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762131 6319 scope.go:122] "RemoveContainer" containerID="ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2" Nov 03 16:03:20.762366 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.762347 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2\": container with ID starting with ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2 not found: ID does not exist" containerID="ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2" Nov 03 16:03:20.762404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762371 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2"} err="failed to get container status \"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2\": rpc error: code = NotFound desc = could not find container \"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2\": container with ID starting with ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2 not found: ID does not exist" Nov 03 16:03:20.762404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762384 6319 scope.go:122] "RemoveContainer" containerID="00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918" Nov 03 16:03:20.762639 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.762622 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918\": container with ID starting with 00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918 not found: ID does not exist" containerID="00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918" Nov 03 16:03:20.762696 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762646 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918"} err="failed to get container status \"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918\": rpc error: code = NotFound desc = could not find container \"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918\": container with ID starting with 00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918 not found: ID does not exist" Nov 03 16:03:20.762696 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762670 6319 scope.go:122] "RemoveContainer" containerID="1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05" Nov 03 16:03:20.762903 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:20.762888 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05\": container with ID starting with 1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05 not found: ID does not exist" containerID="1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05" Nov 03 16:03:20.762936 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762908 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05"} err="failed to get container status \"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05\": rpc error: code = NotFound desc = could not find container \"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05\": container with ID starting with 1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05 not found: ID does not exist" Nov 03 16:03:20.762936 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.762921 6319 scope.go:122] "RemoveContainer" containerID="e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568" Nov 03 16:03:20.763159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763141 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568"} err="failed to get container status \"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568\": rpc error: code = NotFound desc = could not find container \"e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568\": container with ID starting with e128fa82e97ee6f0709703da10e3b37e77271a2180e2d29dafda6a73f8c72568 not found: ID does not exist" Nov 03 16:03:20.763232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763160 6319 scope.go:122] "RemoveContainer" containerID="761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664" Nov 03 16:03:20.763394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763376 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664"} err="failed to get container status \"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664\": rpc error: code = NotFound desc = could not find container \"761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664\": container with ID starting with 761ef743405d8f62c6afe6888875bd647564521fa91524840d5f1a461e848664 not found: ID does not exist" Nov 03 16:03:20.763436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763395 6319 scope.go:122] "RemoveContainer" containerID="62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210" Nov 03 16:03:20.763597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763580 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210"} err="failed to get container status \"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210\": rpc error: code = NotFound desc = could not find container \"62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210\": container with ID starting with 62c5cbf6364cb18a59b40a4d94693ab7a2a5e7446de098856bc0b32994afc210 not found: ID does not exist" Nov 03 16:03:20.763656 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763598 6319 scope.go:122] "RemoveContainer" containerID="932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8" Nov 03 16:03:20.763817 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763799 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8"} err="failed to get container status \"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8\": rpc error: code = NotFound desc = could not find container \"932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8\": container with ID starting with 932c3b3452426bd1a437cef7e022141fb94e433ba823749bcea79854d3a72dc8 not found: ID does not exist" Nov 03 16:03:20.763884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.763818 6319 scope.go:122] "RemoveContainer" containerID="ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2" Nov 03 16:03:20.764090 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.764056 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2"} err="failed to get container status \"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2\": rpc error: code = NotFound desc = could not find container \"ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2\": container with ID starting with ff0222a2046d02e280b69c681acded188e10d65253750b97512c280441e752e2 not found: ID does not exist" Nov 03 16:03:20.764161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.764091 6319 scope.go:122] "RemoveContainer" containerID="00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918" Nov 03 16:03:20.764382 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.764362 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918"} err="failed to get container status \"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918\": rpc error: code = NotFound desc = could not find container \"00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918\": container with ID starting with 00452cef49b7ea2d9892a57f4dc765fcc824a1c2e3b27ce5caa57f1752c12918 not found: ID does not exist" Nov 03 16:03:20.764382 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.764381 6319 scope.go:122] "RemoveContainer" containerID="1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05" Nov 03 16:03:20.764637 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.764613 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05"} err="failed to get container status \"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05\": rpc error: code = NotFound desc = could not find container \"1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05\": container with ID starting with 1029d87d5ceda89809474b2182b1d5686e7c574039557ca762cc9a3cd4ae8c05 not found: ID does not exist" Nov 03 16:03:20.774231 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774214 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:03:20.774469 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774458 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="init-config-reloader" Nov 03 16:03:20.774507 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774471 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="init-config-reloader" Nov 03 16:03:20.774507 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774494 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-metric" Nov 03 16:03:20.774507 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774499 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-metric" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774511 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="config-reloader" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774516 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="config-reloader" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774522 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="alertmanager" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774526 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="alertmanager" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774536 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774540 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774545 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-web" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774549 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-web" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774557 6319 cpu_manager.go:414] "containerMap: removing container" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="prom-label-proxy" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774561 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="prom-label-proxy" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774607 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-metric" Nov 03 16:03:20.774609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774614 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="config-reloader" Nov 03 16:03:20.774933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774621 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="alertmanager" Nov 03 16:03:20.774933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774627 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy" Nov 03 16:03:20.774933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774631 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="kube-rbac-proxy-web" Nov 03 16:03:20.774933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774636 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="prom-label-proxy" Nov 03 16:03:20.774933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.774641 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" containerName="init-config-reloader" Nov 03 16:03:20.779762 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.779740 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.788169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788144 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 16:03:20.788261 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788187 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 16:03:20.788261 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788202 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 16:03:20.788343 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788312 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 16:03:20.788376 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788343 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 16:03:20.788376 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788370 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 16:03:20.788451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788390 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 16:03:20.788451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788429 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tn6cb\"" Nov 03 16:03:20.788451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.788446 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 16:03:20.791674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.791660 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 16:03:20.796379 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.796359 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:03:20.862597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862572 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00089217-c01a-415e-beb1-b8abc530eb75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862608 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862638 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862689 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00089217-c01a-415e-beb1-b8abc530eb75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862745 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-web-config\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862765 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00089217-c01a-415e-beb1-b8abc530eb75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862790 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrffx\" (UniqueName: \"kubernetes.io/projected/00089217-c01a-415e-beb1-b8abc530eb75-kube-api-access-hrffx\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862915 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862847 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862915 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862880 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.862969 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862911 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00089217-c01a-415e-beb1-b8abc530eb75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.863027 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.862971 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-config-volume\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.863067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.863026 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.863067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.863061 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00089217-c01a-415e-beb1-b8abc530eb75-config-out\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963845 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963816 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00089217-c01a-415e-beb1-b8abc530eb75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963853 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963872 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963888 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00089217-c01a-415e-beb1-b8abc530eb75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963919 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-web-config\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963935 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00089217-c01a-415e-beb1-b8abc530eb75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.963977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.963960 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrffx\" (UniqueName: \"kubernetes.io/projected/00089217-c01a-415e-beb1-b8abc530eb75-kube-api-access-hrffx\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964194 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964268 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964296 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00089217-c01a-415e-beb1-b8abc530eb75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964333 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00089217-c01a-415e-beb1-b8abc530eb75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964324 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-config-volume\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964400 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.964465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.964446 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00089217-c01a-415e-beb1-b8abc530eb75-config-out\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.965334 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.965309 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00089217-c01a-415e-beb1-b8abc530eb75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.966586 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.966562 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00089217-c01a-415e-beb1-b8abc530eb75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.967569 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.967476 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00089217-c01a-415e-beb1-b8abc530eb75-config-out\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.967884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.967720 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-config-volume\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.967884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.967819 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00089217-c01a-415e-beb1-b8abc530eb75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.967884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.967834 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-web-config\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.968097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.967949 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.968097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.968056 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.968185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.968153 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.968852 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.968835 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.969056 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.969038 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00089217-c01a-415e-beb1-b8abc530eb75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:20.977192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:20.977173 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrffx\" (UniqueName: \"kubernetes.io/projected/00089217-c01a-415e-beb1-b8abc530eb75-kube-api-access-hrffx\") pod \"alertmanager-main-0\" (UID: \"00089217-c01a-415e-beb1-b8abc530eb75\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:21.095404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:21.095366 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:03:21.386459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:21.386382 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:03:21.390145 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:03:21.390121 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00089217_c01a_415e_beb1_b8abc530eb75.slice/crio-7cf4dcb25410372d85a91b8b266544435fc33494d9a70ed0979f0f5d8459cb78 WatchSource:0}: Error finding container 7cf4dcb25410372d85a91b8b266544435fc33494d9a70ed0979f0f5d8459cb78: Status 404 returned error can't find the container with id 7cf4dcb25410372d85a91b8b266544435fc33494d9a70ed0979f0f5d8459cb78 Nov 03 16:03:21.718881 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:21.718851 6319 generic.go:356] "Generic (PLEG): container finished" podID="00089217-c01a-415e-beb1-b8abc530eb75" containerID="f8450c62f17ca614e04c0eadd18fd78925bdf1020e3dc8cf278cfae988789f87" exitCode=0 Nov 03 16:03:21.719263 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:21.718891 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerDied","Data":"f8450c62f17ca614e04c0eadd18fd78925bdf1020e3dc8cf278cfae988789f87"} Nov 03 16:03:21.719263 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:21.718942 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"7cf4dcb25410372d85a91b8b266544435fc33494d9a70ed0979f0f5d8459cb78"} Nov 03 16:03:21.755612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:21.755580 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-b64r4" Nov 03 16:03:22.641088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.641054 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7540b6-fcc5-4561-bc49-8ccfd4a8b616" path="/var/lib/kubelet/pods/af7540b6-fcc5-4561-bc49-8ccfd4a8b616/volumes" Nov 03 16:03:22.641576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.641564 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:22.727053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.727019 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"daedeae57e1459643be4ed9499ca43ef51ab670b8f42ab2e6ddc39c049d7737d"} Nov 03 16:03:22.727386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.727097 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"d6a8aea7599003a7e9a5d24346118e711064f667d5252e841ba1a4e28684d4f5"} Nov 03 16:03:22.727386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.727136 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"658d39294e8f5e49344800a560c3195dcb4ecff51b8097f34f07f648d3346a55"} Nov 03 16:03:22.727386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.727171 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"ae352e3b245201b6a8bfc57efd76a3f71f624f8c0e4080a11bb74ab6c5fbe14b"} Nov 03 16:03:22.727386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.727209 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"770700dc7254a785f3744c360c76ad2edcc8617105ed017e7cae055b6e16f54b"} Nov 03 16:03:22.727386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.727241 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00089217-c01a-415e-beb1-b8abc530eb75","Type":"ContainerStarted","Data":"3883eeb3ac3a70786783394793e3675ff46f35dc043520befb89da4cae9f02ee"} Nov 03 16:03:22.761554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.761504 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.743396838 podStartE2EDuration="2.761491081s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.094243ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:03:22.760136575 +0000 UTC m=+6.271504444" watchObservedRunningTime="2026-11-03 16:03:22.761491081 +0000 UTC m=+6.272858961" Nov 03 16:03:22.791528 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.791501 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:22.796026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.796004 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:22.861779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:22.861752 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:23.283258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:23.283223 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xmqfv" Nov 03 16:03:23.512712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:23.512687 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:26.093623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:26.093532 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:26.094010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:26.093679 6319 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:03:26.097463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:26.097439 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565565c4b5-zqhq2" Nov 03 16:03:26.149241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:26.149208 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf46c5f77-bn7bp"] Nov 03 16:03:27.543049 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:27.543019 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:27.543494 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:27.543258 6319 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:03:27.762153 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:27.762119 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:03:41.294384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:41.294356 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7mkd" Nov 03 16:03:42.628517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:42.628455 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bcd5b478b-bwmwn" podUID="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" containerName="console" containerID="cri-o://fcc10e60283abe30ed74a2f360bb5441c969a83768d6e9ff2068ad74790d7f0b" gracePeriod=15 Nov 03 16:03:42.796222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:42.796197 6319 generic.go:356] "Generic (PLEG): container finished" podID="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" containerID="fcc10e60283abe30ed74a2f360bb5441c969a83768d6e9ff2068ad74790d7f0b" exitCode=0 Nov 03 16:03:42.796302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:42.796240 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5b478b-bwmwn" event={"ID":"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e","Type":"ContainerDied","Data":"fcc10e60283abe30ed74a2f360bb5441c969a83768d6e9ff2068ad74790d7f0b"} Nov 03 16:03:42.868702 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:42.868678 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:43.034167 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034133 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034179 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034204 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034231 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034278 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034302 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034362 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") pod \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\" (UID: \"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e\") " Nov 03 16:03:43.034751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034662 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:43.034817 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034695 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:43.034817 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034795 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:43.034912 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.034810 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:43.036792 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.036758 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:43.036888 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.036847 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:43.036888 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.036868 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh" pod "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" (UID: "8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e"). InnerVolumeSpecName "kube-api-access-xtlhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:43.135230 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135188 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtlhh\" (UniqueName: \"kubernetes.io/projected/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-kube-api-access-xtlhh\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.135230 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135222 6319 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-oauth-serving-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.135230 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135234 6319 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-tmp\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.135461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135244 6319 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-serving-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.135461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135254 6319 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-config\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.135461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135264 6319 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.135461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.135272 6319 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e-console-oauth-config\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:43.801467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.801431 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5b478b-bwmwn" event={"ID":"8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e","Type":"ContainerDied","Data":"c23245ec9657513ba00e3e593ab4db2f36a41406c320dabccac998111167bb33"} Nov 03 16:03:43.801467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.801462 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5b478b-bwmwn" Nov 03 16:03:43.801928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.801500 6319 scope.go:122] "RemoveContainer" containerID="fcc10e60283abe30ed74a2f360bb5441c969a83768d6e9ff2068ad74790d7f0b" Nov 03 16:03:43.824242 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.824215 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcd5b478b-bwmwn"] Nov 03 16:03:43.828940 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:43.828920 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bcd5b478b-bwmwn"] Nov 03 16:03:44.640232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:44.640197 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" path="/var/lib/kubelet/pods/8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e/volumes" Nov 03 16:03:51.169029 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.168974 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-cf46c5f77-bn7bp" podUID="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" containerName="console" containerID="cri-o://8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e" gracePeriod=15 Nov 03 16:03:51.410725 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.410702 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:51.596561 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596519 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.596739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596577 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.596739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596613 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.596739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596638 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.596739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596656 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.597005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596749 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.597005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596793 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.597005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596843 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") pod \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\" (UID: \"4213e1e5-022d-4e1a-aa84-f6b0a2c13908\") " Nov 03 16:03:51.597005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.596893 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:03:51.597199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.597088 6319 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-tmp\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.597199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.597149 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:51.597275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.597195 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:51.597275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.597214 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:51.597275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.597244 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:03:51.598963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.598938 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:51.599214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.599185 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "kube-api-access-9w8f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:03:51.599259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.599227 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert" pod "4213e1e5-022d-4e1a-aa84-f6b0a2c13908" (UID: "4213e1e5-022d-4e1a-aa84-f6b0a2c13908"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:03:51.697747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697722 6319 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-serving-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.697747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697745 6319 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-oauth-serving-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.697893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697754 6319 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-oauth-config\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.697893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697764 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w8f9\" (UniqueName: \"kubernetes.io/projected/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-kube-api-access-9w8f9\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.697893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697774 6319 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-trusted-ca-bundle\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.697893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697783 6319 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-console-config\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.697893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.697792 6319 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4213e1e5-022d-4e1a-aa84-f6b0a2c13908-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:03:51.833552 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.833521 6319 generic.go:356] "Generic (PLEG): container finished" podID="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" containerID="8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e" exitCode=0 Nov 03 16:03:51.833714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.833581 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf46c5f77-bn7bp" Nov 03 16:03:51.833714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.833594 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf46c5f77-bn7bp" event={"ID":"4213e1e5-022d-4e1a-aa84-f6b0a2c13908","Type":"ContainerDied","Data":"8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e"} Nov 03 16:03:51.833714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.833654 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf46c5f77-bn7bp" event={"ID":"4213e1e5-022d-4e1a-aa84-f6b0a2c13908","Type":"ContainerDied","Data":"18ef6cad2c8881bc072c6e337be0f5a072f213aa9b10cfddf9c81ed833b9e427"} Nov 03 16:03:51.833714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.833673 6319 scope.go:122] "RemoveContainer" containerID="8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e" Nov 03 16:03:51.841703 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.841688 6319 scope.go:122] "RemoveContainer" containerID="8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e" Nov 03 16:03:51.841974 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:51.841956 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e\": container with ID starting with 8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e not found: ID does not exist" containerID="8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e" Nov 03 16:03:51.842047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.842031 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e"} err="failed to get container status \"8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e\": rpc error: code = NotFound desc = could not find container \"8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e\": container with ID starting with 8beb5561ef0ff7983b7d0a0b76e3a844b6cedf46c2e64db8a2b66b9a5f9cce3e not found: ID does not exist" Nov 03 16:03:51.855026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.854956 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf46c5f77-bn7bp"] Nov 03 16:03:51.859538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:51.859516 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cf46c5f77-bn7bp"] Nov 03 16:03:52.640122 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:52.640090 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" path="/var/lib/kubelet/pods/4213e1e5-022d-4e1a-aa84-f6b0a2c13908/volumes" Nov 03 16:03:52.644192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:03:52.644162 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n6lcf" Nov 03 16:03:57.760643 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:03:57.760611 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:04:27.761449 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:04:27.761412 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:04:57.761272 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:04:57.761193 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:05:27.759713 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:05:27.759677 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:05:57.760873 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:05:57.760837 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:06:27.761656 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:06:27.761581 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:06:57.761320 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:06:57.761287 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:07:27.761029 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:07:27.760995 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:07:57.761089 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:07:57.761007 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:08:27.761091 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:08:27.761051 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:08:57.761331 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:08:57.761294 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:09:27.761037 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:09:27.760940 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:09:57.759801 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:09:57.759765 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:10:27.760425 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:10:27.760389 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:10:29.868465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868438 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn"] Nov 03 16:10:29.868832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868678 6319 cpu_manager.go:414] "containerMap: removing container" podUID="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" containerName="console" Nov 03 16:10:29.868832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868688 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" containerName="console" Nov 03 16:10:29.868832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868703 6319 cpu_manager.go:414] "containerMap: removing container" podUID="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" containerName="console" Nov 03 16:10:29.868832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868708 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" containerName="console" Nov 03 16:10:29.868832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868772 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="4213e1e5-022d-4e1a-aa84-f6b0a2c13908" containerName="console" Nov 03 16:10:29.868832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.868780 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5d7c35-6d31-483d-b2fb-13cf98fd4b6e" containerName="console" Nov 03 16:10:29.871755 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.871735 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:29.878254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.878231 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 16:10:29.878353 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.878231 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 16:10:29.888109 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.888091 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-f4nqg\"" Nov 03 16:10:29.895226 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.895207 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn"] Nov 03 16:10:29.950168 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.950142 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs"] Nov 03 16:10:29.953266 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.953250 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:29.966835 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.966805 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs"] Nov 03 16:10:29.975658 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:29.975637 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z56d6\"" Nov 03 16:10:30.002961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.002931 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.003134 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.002973 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.003134 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.003086 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47bq\" (UniqueName: \"kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.074872 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.074838 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2"] Nov 03 16:10:30.078472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.078449 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.083846 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.083821 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 16:10:30.084082 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.084054 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 16:10:30.084184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.084096 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 16:10:30.084184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.084056 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 16:10:30.084184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.084164 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 16:10:30.084330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.084234 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6779w\"" Nov 03 16:10:30.086077 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.086050 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 16:10:30.090680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.090659 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2"] Nov 03 16:10:30.103603 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103581 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.103720 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103625 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.103720 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103667 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47bq\" (UniqueName: \"kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.103841 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103764 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.103841 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103811 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.103944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103841 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkq9w\" (UniqueName: \"kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.103944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.103925 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.104169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.104150 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.121172 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.121119 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47bq\" (UniqueName: \"kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.187091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.187068 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:30.205124 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205102 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.205229 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205138 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqmq\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.205229 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205169 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.205229 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205189 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkq9w\" (UniqueName: \"kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.205356 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205335 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.205399 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205355 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.205399 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205374 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.205545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205503 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.205605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205553 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.205605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205565 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.205690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205643 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.205690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.205639 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.217751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.217727 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkq9w\" (UniqueName: \"kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.268254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.268225 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:30.306328 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306288 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306350 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqmq\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306398 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306440 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306466 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306519 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306556 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.306996 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.306958 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.309036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.309012 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.309151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.309101 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.309241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.309222 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.309622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.309603 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.320357 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.320305 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqmq\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.320790 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.320712 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-w4zw2\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.391152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.391088 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:30.463407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.463355 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn"] Nov 03 16:10:30.465935 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:10:30.465910 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b3ec69_fa93_4328_a3e0_1f8edc1f4d5d.slice/crio-53832de2cf2f64c66241495765ba0262ff95bee7f755e7c99b0a4a97d8bf8f19 WatchSource:0}: Error finding container 53832de2cf2f64c66241495765ba0262ff95bee7f755e7c99b0a4a97d8bf8f19: Status 404 returned error can't find the container with id 53832de2cf2f64c66241495765ba0262ff95bee7f755e7c99b0a4a97d8bf8f19 Nov 03 16:10:30.467910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.467891 6319 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:10:30.543464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.543436 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs"] Nov 03 16:10:30.546909 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:10:30.546885 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dd586d_64f0_44b6_8207_5b87822463f6.slice/crio-f2c41bb7c3801de32c232da9799046773e288e72dadc7955b5e9f51222f2f859 WatchSource:0}: Error finding container f2c41bb7c3801de32c232da9799046773e288e72dadc7955b5e9f51222f2f859: Status 404 returned error can't find the container with id f2c41bb7c3801de32c232da9799046773e288e72dadc7955b5e9f51222f2f859 Nov 03 16:10:30.667260 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:30.667180 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2"] Nov 03 16:10:30.670640 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:10:30.670605 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54799d3_2eee_4ad3_ab1b_931b3dbcfb20.slice/crio-21ba5bc2baf3e0e8ab49c817edba6167ee90cd440e4c9df02ff069772b7c7b62 WatchSource:0}: Error finding container 21ba5bc2baf3e0e8ab49c817edba6167ee90cd440e4c9df02ff069772b7c7b62: Status 404 returned error can't find the container with id 21ba5bc2baf3e0e8ab49c817edba6167ee90cd440e4c9df02ff069772b7c7b62 Nov 03 16:10:31.125500 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:31.125460 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" event={"ID":"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d","Type":"ContainerStarted","Data":"53832de2cf2f64c66241495765ba0262ff95bee7f755e7c99b0a4a97d8bf8f19"} Nov 03 16:10:31.127111 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:31.127084 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" event={"ID":"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20","Type":"ContainerStarted","Data":"21ba5bc2baf3e0e8ab49c817edba6167ee90cd440e4c9df02ff069772b7c7b62"} Nov 03 16:10:31.128468 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:31.128441 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" event={"ID":"26dd586d-64f0-44b6-8207-5b87822463f6","Type":"ContainerStarted","Data":"f2c41bb7c3801de32c232da9799046773e288e72dadc7955b5e9f51222f2f859"} Nov 03 16:10:34.606926 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:34.606886 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:10:34.607279 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:34.606977 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:10:35.152461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:35.152416 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" event={"ID":"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20","Type":"ContainerStarted","Data":"d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24"} Nov 03 16:10:35.153404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:35.153381 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:35.155240 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:35.155195 6319 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-w4zw2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:10:35.155732 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:35.155437 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" podUID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:10:35.178332 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:35.178278 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" podStartSLOduration=1.245070063 podStartE2EDuration="5.178262653s" totalImagesPullingTime="3.93319259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:35.176696987 +0000 UTC m=+438.688064881" watchObservedRunningTime="2026-11-03 16:10:35.178262653 +0000 UTC m=+438.689630521" Nov 03 16:10:36.158157 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:36.158128 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:10:38.166191 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:38.166107 6319 generic.go:356] "Generic (PLEG): container finished" podID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerID="c7d7f63f078ff831f4f430d562689c98292f24f2eed0538f7df7855edba8751a" exitCode=0 Nov 03 16:10:38.166615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:38.166194 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" event={"ID":"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d","Type":"ContainerDied","Data":"c7d7f63f078ff831f4f430d562689c98292f24f2eed0538f7df7855edba8751a"} Nov 03 16:10:38.167668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:38.167643 6319 generic.go:356] "Generic (PLEG): container finished" podID="26dd586d-64f0-44b6-8207-5b87822463f6" containerID="e2a4380ada73df92496d65d20341efada5bc7a6ae83a1e52a2bee1e9cf6ad03b" exitCode=0 Nov 03 16:10:38.167805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:38.167699 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" event={"ID":"26dd586d-64f0-44b6-8207-5b87822463f6","Type":"ContainerDied","Data":"e2a4380ada73df92496d65d20341efada5bc7a6ae83a1e52a2bee1e9cf6ad03b"} Nov 03 16:10:42.186102 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:42.186062 6319 generic.go:356] "Generic (PLEG): container finished" podID="26dd586d-64f0-44b6-8207-5b87822463f6" containerID="cd623be2e8a15b2d953117040409930f5d10b67ca0aeacb99f7b7d33814b742d" exitCode=0 Nov 03 16:10:42.186554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:42.186143 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" event={"ID":"26dd586d-64f0-44b6-8207-5b87822463f6","Type":"ContainerDied","Data":"cd623be2e8a15b2d953117040409930f5d10b67ca0aeacb99f7b7d33814b742d"} Nov 03 16:10:42.188070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:42.188010 6319 generic.go:356] "Generic (PLEG): container finished" podID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerID="969160fcfb61958c667fe71d6c108ef99b5712ec8067bf04f56f199c10d5325d" exitCode=0 Nov 03 16:10:42.188070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:42.188063 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" event={"ID":"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d","Type":"ContainerDied","Data":"969160fcfb61958c667fe71d6c108ef99b5712ec8067bf04f56f199c10d5325d"} Nov 03 16:10:49.218398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:49.218368 6319 generic.go:356] "Generic (PLEG): container finished" podID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerID="81caf2e6d34703716febff82d258c15333c1ae530101a41a77c6a9e6cacb9dcb" exitCode=0 Nov 03 16:10:49.218775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:49.218497 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" event={"ID":"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d","Type":"ContainerDied","Data":"81caf2e6d34703716febff82d258c15333c1ae530101a41a77c6a9e6cacb9dcb"} Nov 03 16:10:49.220425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:49.220399 6319 generic.go:356] "Generic (PLEG): container finished" podID="26dd586d-64f0-44b6-8207-5b87822463f6" containerID="b652909c8942d67ebec83a1fb4aa778e28e3bc58fe3fc8a721318c3d0f05b688" exitCode=0 Nov 03 16:10:49.220526 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:49.220439 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" event={"ID":"26dd586d-64f0-44b6-8207-5b87822463f6","Type":"ContainerDied","Data":"b652909c8942d67ebec83a1fb4aa778e28e3bc58fe3fc8a721318c3d0f05b688"} Nov 03 16:10:50.353530 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.353501 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:50.371669 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.371650 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:50.387837 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.387813 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util\") pod \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " Nov 03 16:10:50.387958 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.387877 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle\") pod \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " Nov 03 16:10:50.387958 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.387898 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq\" (UniqueName: \"kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq\") pod \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\" (UID: \"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d\") " Nov 03 16:10:50.388338 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.388316 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle" pod "57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" (UID: "57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:10:50.390276 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.390253 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq" pod "57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" (UID: "57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d"). InnerVolumeSpecName "kube-api-access-b47bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:10:50.393045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.393023 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util" pod "57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" (UID: "57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:10:50.488730 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.488655 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle\") pod \"26dd586d-64f0-44b6-8207-5b87822463f6\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " Nov 03 16:10:50.488730 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.488697 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w\" (UniqueName: \"kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w\") pod \"26dd586d-64f0-44b6-8207-5b87822463f6\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " Nov 03 16:10:50.488896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.488749 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util\") pod \"26dd586d-64f0-44b6-8207-5b87822463f6\" (UID: \"26dd586d-64f0-44b6-8207-5b87822463f6\") " Nov 03 16:10:50.488896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.488866 6319 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-bundle\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:10:50.488896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.488877 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47bq\" (UniqueName: \"kubernetes.io/projected/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-kube-api-access-b47bq\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:10:50.488896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.488891 6319 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d-util\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:10:50.489423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.489399 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle" pod "26dd586d-64f0-44b6-8207-5b87822463f6" (UID: "26dd586d-64f0-44b6-8207-5b87822463f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:10:50.490883 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.490864 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w" pod "26dd586d-64f0-44b6-8207-5b87822463f6" (UID: "26dd586d-64f0-44b6-8207-5b87822463f6"). InnerVolumeSpecName "kube-api-access-hkq9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:10:50.492669 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.492648 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util" pod "26dd586d-64f0-44b6-8207-5b87822463f6" (UID: "26dd586d-64f0-44b6-8207-5b87822463f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:10:50.589783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.589757 6319 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-bundle\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:10:50.589783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.589782 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkq9w\" (UniqueName: \"kubernetes.io/projected/26dd586d-64f0-44b6-8207-5b87822463f6-kube-api-access-hkq9w\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:10:50.589783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:50.589791 6319 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26dd586d-64f0-44b6-8207-5b87822463f6-util\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:10:51.229216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:51.229180 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" Nov 03 16:10:51.229406 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:51.229184 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6nnhs" event={"ID":"26dd586d-64f0-44b6-8207-5b87822463f6","Type":"ContainerDied","Data":"f2c41bb7c3801de32c232da9799046773e288e72dadc7955b5e9f51222f2f859"} Nov 03 16:10:51.229406 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:51.229339 6319 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2c41bb7c3801de32c232da9799046773e288e72dadc7955b5e9f51222f2f859" Nov 03 16:10:51.231531 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:51.231312 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" event={"ID":"57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d","Type":"ContainerDied","Data":"53832de2cf2f64c66241495765ba0262ff95bee7f755e7c99b0a4a97d8bf8f19"} Nov 03 16:10:51.231531 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:51.231360 6319 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53832de2cf2f64c66241495765ba0262ff95bee7f755e7c99b0a4a97d8bf8f19" Nov 03 16:10:51.231531 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:10:51.231383 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644c7fwn" Nov 03 16:10:57.762005 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:10:57.761914 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:11:12.508484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508450 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd"] Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508743 6319 cpu_manager.go:414] "containerMap: removing container" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="pull" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508754 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="pull" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508766 6319 cpu_manager.go:414] "containerMap: removing container" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="util" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508771 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="util" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508782 6319 cpu_manager.go:414] "containerMap: removing container" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="util" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508787 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="util" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508793 6319 cpu_manager.go:414] "containerMap: removing container" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="extract" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508797 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="extract" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508803 6319 cpu_manager.go:414] "containerMap: removing container" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="extract" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508807 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="extract" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508817 6319 cpu_manager.go:414] "containerMap: removing container" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="pull" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508821 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="pull" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508903 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="26dd586d-64f0-44b6-8207-5b87822463f6" containerName="extract" Nov 03 16:11:12.508921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.508910 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b3ec69-fa93-4328-a3e0-1f8edc1f4d5d" containerName="extract" Nov 03 16:11:12.510919 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.510897 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.546822 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.546777 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd"] Nov 03 16:11:12.590322 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.590294 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 16:11:12.590451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.590337 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 16:11:12.590515 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.590500 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 16:11:12.590555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.590526 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 16:11:12.590600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.590570 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 16:11:12.590773 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.590758 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-f626w\"" Nov 03 16:11:12.655071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.655030 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4lvx\" (UniqueName: \"kubernetes.io/projected/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-kube-api-access-h4lvx\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.655214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.655085 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.655214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.655164 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.655214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.655197 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.655317 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.655215 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.655350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.655316 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.755746 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.755713 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.755746 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.755747 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.755960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.755800 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.755960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.755822 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4lvx\" (UniqueName: \"kubernetes.io/projected/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-kube-api-access-h4lvx\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.755960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.755867 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.755960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.755903 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.756371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.756345 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-manifests\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.758551 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.758487 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-tmp\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.758650 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.758570 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.758650 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.758611 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-metrics-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.758760 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.758746 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.769223 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.769199 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4lvx\" (UniqueName: \"kubernetes.io/projected/b4fddbce-22a2-40c8-876a-3f5baeb36b4f-kube-api-access-h4lvx\") pod \"opendatahub-operator-controller-manager-d59cb86d5-46fvd\" (UID: \"b4fddbce-22a2-40c8-876a-3f5baeb36b4f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:12.824247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:12.824219 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:13.132005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:13.131908 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd"] Nov 03 16:11:13.135211 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:11:13.135181 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fddbce_22a2_40c8_876a_3f5baeb36b4f.slice/crio-0b2f8919ecedfb866d125c075d2c6030fd425f0be874a2ab7e0dd79f14f022d4 WatchSource:0}: Error finding container 0b2f8919ecedfb866d125c075d2c6030fd425f0be874a2ab7e0dd79f14f022d4: Status 404 returned error can't find the container with id 0b2f8919ecedfb866d125c075d2c6030fd425f0be874a2ab7e0dd79f14f022d4 Nov 03 16:11:13.318443 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:13.318409 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" event={"ID":"b4fddbce-22a2-40c8-876a-3f5baeb36b4f","Type":"ContainerStarted","Data":"0b2f8919ecedfb866d125c075d2c6030fd425f0be874a2ab7e0dd79f14f022d4"} Nov 03 16:11:16.334180 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:16.334109 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" event={"ID":"b4fddbce-22a2-40c8-876a-3f5baeb36b4f","Type":"ContainerStarted","Data":"e40b6d25b27169b4c22e063f2c6efe26f9385807cbd8b9229a030372bc40dbce"} Nov 03 16:11:17.339214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:17.339174 6319 generic.go:356] "Generic (PLEG): container finished" podID="b4fddbce-22a2-40c8-876a-3f5baeb36b4f" containerID="e40b6d25b27169b4c22e063f2c6efe26f9385807cbd8b9229a030372bc40dbce" exitCode=0 Nov 03 16:11:17.339591 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:17.339263 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" event={"ID":"b4fddbce-22a2-40c8-876a-3f5baeb36b4f","Type":"ContainerDied","Data":"e40b6d25b27169b4c22e063f2c6efe26f9385807cbd8b9229a030372bc40dbce"} Nov 03 16:11:18.345001 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:18.344943 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" event={"ID":"b4fddbce-22a2-40c8-876a-3f5baeb36b4f","Type":"ContainerStarted","Data":"846a65f92080188f4d39b00dc52eb67d18849c4c68bc944651fa00c06693eaf1"} Nov 03 16:11:18.345450 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:18.345062 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:27.761904 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:11:27.761872 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:11:29.351910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:29.351881 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" Nov 03 16:11:29.397887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:29.397847 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d59cb86d5-46fvd" podStartSLOduration=13.706889447 podStartE2EDuration="17.397835685s" totalImagesPullingTime="3.281706676s" totalInitContainerRuntime="409.239562ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:18.478473507 +0000 UTC m=+481.989841374" watchObservedRunningTime="2026-11-03 16:11:29.397835685 +0000 UTC m=+492.909203553" Nov 03 16:11:54.445865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.445837 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg"] Nov 03 16:11:54.451545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.451529 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.475870 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.475844 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg"] Nov 03 16:11:54.475870 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.475859 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 16:11:54.476088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.475965 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 16:11:54.476156 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.476106 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wdnhl\"" Nov 03 16:11:54.476208 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.476155 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 16:11:54.483219 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.483203 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 16:11:54.495051 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.494976 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/89ed5768-10fd-4495-8be7-559b43295d93-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.495166 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.495114 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/89ed5768-10fd-4495-8be7-559b43295d93-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.495253 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.495222 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fvc\" (UniqueName: \"kubernetes.io/projected/89ed5768-10fd-4495-8be7-559b43295d93-kube-api-access-m9fvc\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.596055 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.596021 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/89ed5768-10fd-4495-8be7-559b43295d93-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.596194 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.596074 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fvc\" (UniqueName: \"kubernetes.io/projected/89ed5768-10fd-4495-8be7-559b43295d93-kube-api-access-m9fvc\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.596194 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.596118 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/89ed5768-10fd-4495-8be7-559b43295d93-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.596669 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.596650 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/89ed5768-10fd-4495-8be7-559b43295d93-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.598572 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.598552 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/89ed5768-10fd-4495-8be7-559b43295d93-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.629169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.629138 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fvc\" (UniqueName: \"kubernetes.io/projected/89ed5768-10fd-4495-8be7-559b43295d93-kube-api-access-m9fvc\") pod \"kuadrant-console-plugin-66c4fb575d-tlqlg\" (UID: \"89ed5768-10fd-4495-8be7-559b43295d93\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.641861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.641839 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6"] Nov 03 16:11:54.644119 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.644101 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.662111 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.662089 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6"] Nov 03 16:11:54.668227 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.668205 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 16:11:54.668350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.668276 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 16:11:54.668350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.668330 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z9774\"" Nov 03 16:11:54.696965 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.696902 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfdg\" (UniqueName: \"kubernetes.io/projected/244e1060-bcfe-466b-89cf-e85f9e1429a0-kube-api-access-ndfdg\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.696965 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.696954 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244e1060-bcfe-466b-89cf-e85f9e1429a0-tls-certs\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.697137 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.697010 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/244e1060-bcfe-466b-89cf-e85f9e1429a0-tmp\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.779200 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.779163 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" Nov 03 16:11:54.797686 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.797661 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244e1060-bcfe-466b-89cf-e85f9e1429a0-tls-certs\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.797775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.797701 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/244e1060-bcfe-466b-89cf-e85f9e1429a0-tmp\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.797815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.797769 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfdg\" (UniqueName: \"kubernetes.io/projected/244e1060-bcfe-466b-89cf-e85f9e1429a0-kube-api-access-ndfdg\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.800056 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.800025 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/244e1060-bcfe-466b-89cf-e85f9e1429a0-tmp\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.800488 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.800466 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244e1060-bcfe-466b-89cf-e85f9e1429a0-tls-certs\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.838488 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.838466 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfdg\" (UniqueName: \"kubernetes.io/projected/244e1060-bcfe-466b-89cf-e85f9e1429a0-kube-api-access-ndfdg\") pod \"kube-auth-proxy-864dbfdb86-8lnw6\" (UID: \"244e1060-bcfe-466b-89cf-e85f9e1429a0\") " pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:54.955035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:54.954918 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" Nov 03 16:11:55.054032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:55.054008 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg"] Nov 03 16:11:55.057020 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:11:55.056964 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89ed5768_10fd_4495_8be7_559b43295d93.slice/crio-6de84a4357bf28d646806931e2322165a8a80e834ae0144e4cad6c502cac06f6 WatchSource:0}: Error finding container 6de84a4357bf28d646806931e2322165a8a80e834ae0144e4cad6c502cac06f6: Status 404 returned error can't find the container with id 6de84a4357bf28d646806931e2322165a8a80e834ae0144e4cad6c502cac06f6 Nov 03 16:11:55.224765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:55.224735 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6"] Nov 03 16:11:55.227857 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:11:55.227830 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244e1060_bcfe_466b_89cf_e85f9e1429a0.slice/crio-8e6b13be1f0d4efde3dc8c092ae3de8e68c98c5f3a7da0ce7891c17c64a140a2 WatchSource:0}: Error finding container 8e6b13be1f0d4efde3dc8c092ae3de8e68c98c5f3a7da0ce7891c17c64a140a2: Status 404 returned error can't find the container with id 8e6b13be1f0d4efde3dc8c092ae3de8e68c98c5f3a7da0ce7891c17c64a140a2 Nov 03 16:11:55.482652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:55.482560 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" event={"ID":"244e1060-bcfe-466b-89cf-e85f9e1429a0","Type":"ContainerStarted","Data":"8e6b13be1f0d4efde3dc8c092ae3de8e68c98c5f3a7da0ce7891c17c64a140a2"} Nov 03 16:11:55.483707 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:55.483686 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" event={"ID":"89ed5768-10fd-4495-8be7-559b43295d93","Type":"ContainerStarted","Data":"6de84a4357bf28d646806931e2322165a8a80e834ae0144e4cad6c502cac06f6"} Nov 03 16:11:57.772271 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:11:57.772233 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:11:58.502047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:58.502009 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" event={"ID":"244e1060-bcfe-466b-89cf-e85f9e1429a0","Type":"ContainerStarted","Data":"307c386ae2f889a68b57232449b1b69c2536ffb3413b70aac380d285bd14f0f4"} Nov 03 16:11:58.531402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:11:58.531334 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-864dbfdb86-8lnw6" podStartSLOduration=1.643435137 podStartE2EDuration="4.531314714s" totalImagesPullingTime="2.887879577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:58.530264766 +0000 UTC m=+522.041632637" watchObservedRunningTime="2026-11-03 16:11:58.531314714 +0000 UTC m=+522.042682585" Nov 03 16:12:00.512049 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:00.512017 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" event={"ID":"89ed5768-10fd-4495-8be7-559b43295d93","Type":"ContainerStarted","Data":"55e0eeaa1297d696e96db6fec4ab69377a20393ff0a775afe2893d348638852c"} Nov 03 16:12:00.532531 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:00.532483 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tlqlg" podStartSLOduration=1.404819551 podStartE2EDuration="6.532468742s" totalImagesPullingTime="5.127649191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:00.531773773 +0000 UTC m=+524.043141645" watchObservedRunningTime="2026-11-03 16:12:00.532468742 +0000 UTC m=+524.043836610" Nov 03 16:12:27.762562 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:12:27.762437 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:12:51.023477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.023447 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ntj89"] Nov 03 16:12:51.026385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.026367 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.033670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.033652 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 16:12:51.033750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.033676 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 16:12:51.039362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.039332 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ntj89"] Nov 03 16:12:51.039459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.039423 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-chpgk\"" Nov 03 16:12:51.039572 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.039554 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 16:12:51.079540 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.079514 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-j7zvw"] Nov 03 16:12:51.082336 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.082323 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.097632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.097606 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-j7zvw"] Nov 03 16:12:51.103410 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.103384 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-bjn8p"] Nov 03 16:12:51.106267 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.106251 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.117864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.117842 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fvk6l\"" Nov 03 16:12:51.117967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.117883 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 16:12:51.121309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.121289 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bjn8p"] Nov 03 16:12:51.132651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.132630 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 16:12:51.132785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.132702 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 16:12:51.132785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.132709 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9kfgj\"" Nov 03 16:12:51.158235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158206 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b9503-cf4d-4749-94b7-eb4101c00c59-tls-certs\") pod \"model-serving-api-75bd89fdfc-j7zvw\" (UID: \"440b9503-cf4d-4749-94b7-eb4101c00c59\") " pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.158392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158246 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zxg\" (UniqueName: \"kubernetes.io/projected/b78b08a9-11af-446a-abce-551fc42a58e6-kube-api-access-l6zxg\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.158392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158270 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4827a0-953e-4290-8c36-a33fe77091bd-cert\") pod \"kserve-controller-manager-776cc66d47-ntj89\" (UID: \"7a4827a0-953e-4290-8c36-a33fe77091bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.158392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158379 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkjq\" (UniqueName: \"kubernetes.io/projected/7a4827a0-953e-4290-8c36-a33fe77091bd-kube-api-access-dbkjq\") pod \"kserve-controller-manager-776cc66d47-ntj89\" (UID: \"7a4827a0-953e-4290-8c36-a33fe77091bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.158534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158401 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6crp\" (UniqueName: \"kubernetes.io/projected/440b9503-cf4d-4749-94b7-eb4101c00c59-kube-api-access-b6crp\") pod \"model-serving-api-75bd89fdfc-j7zvw\" (UID: \"440b9503-cf4d-4749-94b7-eb4101c00c59\") " pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.158534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158438 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b78b08a9-11af-446a-abce-551fc42a58e6-cert\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.158534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.158519 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78b08a9-11af-446a-abce-551fc42a58e6-metrics-certs\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.259612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259577 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b9503-cf4d-4749-94b7-eb4101c00c59-tls-certs\") pod \"model-serving-api-75bd89fdfc-j7zvw\" (UID: \"440b9503-cf4d-4749-94b7-eb4101c00c59\") " pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.259775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259618 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zxg\" (UniqueName: \"kubernetes.io/projected/b78b08a9-11af-446a-abce-551fc42a58e6-kube-api-access-l6zxg\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.259775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259646 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4827a0-953e-4290-8c36-a33fe77091bd-cert\") pod \"kserve-controller-manager-776cc66d47-ntj89\" (UID: \"7a4827a0-953e-4290-8c36-a33fe77091bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.259775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259676 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkjq\" (UniqueName: \"kubernetes.io/projected/7a4827a0-953e-4290-8c36-a33fe77091bd-kube-api-access-dbkjq\") pod \"kserve-controller-manager-776cc66d47-ntj89\" (UID: \"7a4827a0-953e-4290-8c36-a33fe77091bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.259775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259707 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6crp\" (UniqueName: \"kubernetes.io/projected/440b9503-cf4d-4749-94b7-eb4101c00c59-kube-api-access-b6crp\") pod \"model-serving-api-75bd89fdfc-j7zvw\" (UID: \"440b9503-cf4d-4749-94b7-eb4101c00c59\") " pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.259775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259749 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b78b08a9-11af-446a-abce-551fc42a58e6-cert\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.260100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.259783 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78b08a9-11af-446a-abce-551fc42a58e6-metrics-certs\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.262373 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.262347 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b78b08a9-11af-446a-abce-551fc42a58e6-cert\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.262490 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.262432 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78b08a9-11af-446a-abce-551fc42a58e6-metrics-certs\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.262490 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.262441 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a4827a0-953e-4290-8c36-a33fe77091bd-cert\") pod \"kserve-controller-manager-776cc66d47-ntj89\" (UID: \"7a4827a0-953e-4290-8c36-a33fe77091bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.262569 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.262510 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440b9503-cf4d-4749-94b7-eb4101c00c59-tls-certs\") pod \"model-serving-api-75bd89fdfc-j7zvw\" (UID: \"440b9503-cf4d-4749-94b7-eb4101c00c59\") " pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.278642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.278584 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6crp\" (UniqueName: \"kubernetes.io/projected/440b9503-cf4d-4749-94b7-eb4101c00c59-kube-api-access-b6crp\") pod \"model-serving-api-75bd89fdfc-j7zvw\" (UID: \"440b9503-cf4d-4749-94b7-eb4101c00c59\") " pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.279467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.279445 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zxg\" (UniqueName: \"kubernetes.io/projected/b78b08a9-11af-446a-abce-551fc42a58e6-kube-api-access-l6zxg\") pod \"odh-model-controller-769875db56-bjn8p\" (UID: \"b78b08a9-11af-446a-abce-551fc42a58e6\") " pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.279525 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.279506 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkjq\" (UniqueName: \"kubernetes.io/projected/7a4827a0-953e-4290-8c36-a33fe77091bd-kube-api-access-dbkjq\") pod \"kserve-controller-manager-776cc66d47-ntj89\" (UID: \"7a4827a0-953e-4290-8c36-a33fe77091bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.339446 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.339425 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:51.394316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.394285 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:51.417584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.417538 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:51.530970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.530899 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7"] Nov 03 16:12:51.534208 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.534192 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.550595 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.550572 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7"] Nov 03 16:12:51.612854 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.612826 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ntj89"] Nov 03 16:12:51.615705 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:12:51.615680 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4827a0_953e_4290_8c36_a33fe77091bd.slice/crio-d996f298bf6336aff3a1405d4b57a1097714bea30101872bce59235a3891102f WatchSource:0}: Error finding container d996f298bf6336aff3a1405d4b57a1097714bea30101872bce59235a3891102f: Status 404 returned error can't find the container with id d996f298bf6336aff3a1405d4b57a1097714bea30101872bce59235a3891102f Nov 03 16:12:51.666105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666083 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.666215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666111 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.666215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666134 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.666294 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666278 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace6aed5-9f03-485c-84cc-fcf08239b44a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.666333 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666304 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.666367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666328 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsc8b\" (UniqueName: \"kubernetes.io/projected/ace6aed5-9f03-485c-84cc-fcf08239b44a-kube-api-access-tsc8b\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.666444 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.666425 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.694311 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.694290 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-j7zvw"] Nov 03 16:12:51.697276 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:12:51.697254 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440b9503_cf4d_4749_94b7_eb4101c00c59.slice/crio-ed4258b9c3ad75ccc1699e4d5f959f6cf033223e0d8231930fac1950f84ab1b2 WatchSource:0}: Error finding container ed4258b9c3ad75ccc1699e4d5f959f6cf033223e0d8231930fac1950f84ab1b2: Status 404 returned error can't find the container with id ed4258b9c3ad75ccc1699e4d5f959f6cf033223e0d8231930fac1950f84ab1b2 Nov 03 16:12:51.713239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.713217 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" event={"ID":"440b9503-cf4d-4749-94b7-eb4101c00c59","Type":"ContainerStarted","Data":"ed4258b9c3ad75ccc1699e4d5f959f6cf033223e0d8231930fac1950f84ab1b2"} Nov 03 16:12:51.714250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.714230 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" event={"ID":"7a4827a0-953e-4290-8c36-a33fe77091bd","Type":"ContainerStarted","Data":"d996f298bf6336aff3a1405d4b57a1097714bea30101872bce59235a3891102f"} Nov 03 16:12:51.724611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.724590 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bjn8p"] Nov 03 16:12:51.727420 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:12:51.727397 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78b08a9_11af_446a_abce_551fc42a58e6.slice/crio-f90d0d8678dd3133a579299bb49a392618e5a438009a53eabfbd474f56f7b401 WatchSource:0}: Error finding container f90d0d8678dd3133a579299bb49a392618e5a438009a53eabfbd474f56f7b401: Status 404 returned error can't find the container with id f90d0d8678dd3133a579299bb49a392618e5a438009a53eabfbd474f56f7b401 Nov 03 16:12:51.766924 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.766906 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace6aed5-9f03-485c-84cc-fcf08239b44a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.766930 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.766951 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsc8b\" (UniqueName: \"kubernetes.io/projected/ace6aed5-9f03-485c-84cc-fcf08239b44a-kube-api-access-tsc8b\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767111 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.767075 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.767129 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767195 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.767155 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767195 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.767187 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.767825 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.767757 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.769480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.769458 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.769586 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.769567 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.769802 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.769785 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace6aed5-9f03-485c-84cc-fcf08239b44a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.769912 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.769897 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.779418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.779396 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace6aed5-9f03-485c-84cc-fcf08239b44a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.780022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.779962 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsc8b\" (UniqueName: \"kubernetes.io/projected/ace6aed5-9f03-485c-84cc-fcf08239b44a-kube-api-access-tsc8b\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8ggl7\" (UID: \"ace6aed5-9f03-485c-84cc-fcf08239b44a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:51.846033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:51.845955 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:52.132930 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.132863 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7"] Nov 03 16:12:52.135966 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:12:52.135943 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace6aed5_9f03_485c_84cc_fcf08239b44a.slice/crio-163715fb5057695d32d18cf9b2f8cb28e7b23a157848812b09c3e99124d1297a WatchSource:0}: Error finding container 163715fb5057695d32d18cf9b2f8cb28e7b23a157848812b09c3e99124d1297a: Status 404 returned error can't find the container with id 163715fb5057695d32d18cf9b2f8cb28e7b23a157848812b09c3e99124d1297a Nov 03 16:12:52.138246 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.138211 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:12:52.138329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.138275 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:12:52.726378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.725403 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" event={"ID":"ace6aed5-9f03-485c-84cc-fcf08239b44a","Type":"ContainerStarted","Data":"409c0ecee50ff7fc465a64b8be5fee06e2efe53e415394bde35104613e427e74"} Nov 03 16:12:52.726378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.725478 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" event={"ID":"ace6aed5-9f03-485c-84cc-fcf08239b44a","Type":"ContainerStarted","Data":"163715fb5057695d32d18cf9b2f8cb28e7b23a157848812b09c3e99124d1297a"} Nov 03 16:12:52.726378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.726324 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:52.732958 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.732923 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bjn8p" event={"ID":"b78b08a9-11af-446a-abce-551fc42a58e6","Type":"ContainerStarted","Data":"f90d0d8678dd3133a579299bb49a392618e5a438009a53eabfbd474f56f7b401"} Nov 03 16:12:52.757278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:52.756135 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" podStartSLOduration=1.756109324 podStartE2EDuration="1.756109324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:52.753924888 +0000 UTC m=+576.265293005" watchObservedRunningTime="2026-11-03 16:12:52.756109324 +0000 UTC m=+576.267477203" Nov 03 16:12:53.742436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:53.742245 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8ggl7" Nov 03 16:12:53.812415 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:53.812387 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2"] Nov 03 16:12:53.812627 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:53.812601 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" podUID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" containerName="discovery" containerID="cri-o://d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24" gracePeriod=30 Nov 03 16:12:54.099495 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.099465 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:12:54.193927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.193905 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.194088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.193944 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.194088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.194025 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.194088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.194048 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.194088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.194067 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.194088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.194087 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.194555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.194123 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs\") pod \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\" (UID: \"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20\") " Nov 03 16:12:54.195423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.195353 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:12:54.197799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.197770 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:12:54.198205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.198181 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:12:54.198499 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.198467 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:12:54.198826 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.198801 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "kube-api-access-pnqmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:12:54.199236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.199213 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:12:54.200945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.200911 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts" pod "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" (UID: "e54799d3-2eee-4ad3-ab1b-931b3dbcfb20"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:12:54.295604 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295571 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqmq\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-kube-api-access-pnqmq\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.295604 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295610 6319 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-dns-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.295799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295628 6319 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-kubeconfig\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.295799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295646 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.295799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295662 6319 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-local-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.295799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295679 6319 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-istio-csr-ca-configmap\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.295799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.295694 6319 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20-cacerts\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:12:54.747751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.747716 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" event={"ID":"440b9503-cf4d-4749-94b7-eb4101c00c59","Type":"ContainerStarted","Data":"336de52419d635bc93a492fde7552d237bedd436dc3bf8467bf95a1ecb4a2e63"} Nov 03 16:12:54.748237 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.748019 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:12:54.749709 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.749677 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" event={"ID":"7a4827a0-953e-4290-8c36-a33fe77091bd","Type":"ContainerStarted","Data":"b7736f96e9c74312ee0f8af0762c0cd12c617f75445ec433e4ccf2fd4021a7b7"} Nov 03 16:12:54.749839 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.749772 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:12:54.751162 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.751136 6319 generic.go:356] "Generic (PLEG): container finished" podID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" containerID="d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24" exitCode=0 Nov 03 16:12:54.751275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.751198 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" Nov 03 16:12:54.751275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.751220 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" event={"ID":"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20","Type":"ContainerDied","Data":"d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24"} Nov 03 16:12:54.751275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.751263 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2" event={"ID":"e54799d3-2eee-4ad3-ab1b-931b3dbcfb20","Type":"ContainerDied","Data":"21ba5bc2baf3e0e8ab49c817edba6167ee90cd440e4c9df02ff069772b7c7b62"} Nov 03 16:12:54.751503 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.751289 6319 scope.go:122] "RemoveContainer" containerID="d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24" Nov 03 16:12:54.760966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.760950 6319 scope.go:122] "RemoveContainer" containerID="d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24" Nov 03 16:12:54.761267 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:12:54.761242 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24\": container with ID starting with d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24 not found: ID does not exist" containerID="d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24" Nov 03 16:12:54.761375 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.761275 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24"} err="failed to get container status \"d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24\": rpc error: code = NotFound desc = could not find container \"d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24\": container with ID starting with d86cd40fd033b94a09248ee186afe86b66b6c87717fc440e98c81c16ec12ef24 not found: ID does not exist" Nov 03 16:12:54.769701 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.769657 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" podStartSLOduration=1.2864372259999999 podStartE2EDuration="3.769646028s" totalImagesPullingTime="2.483208802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:54.768272033 +0000 UTC m=+578.279639899" watchObservedRunningTime="2026-11-03 16:12:54.769646028 +0000 UTC m=+578.281013895" Nov 03 16:12:54.804667 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.804622 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" podStartSLOduration=0.762631389 podStartE2EDuration="3.804608514s" totalImagesPullingTime="3.041977125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:54.789187488 +0000 UTC m=+578.300555357" watchObservedRunningTime="2026-11-03 16:12:54.804608514 +0000 UTC m=+578.315976382" Nov 03 16:12:54.805596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.805572 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2"] Nov 03 16:12:54.812610 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:54.812590 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-w4zw2"] Nov 03 16:12:55.756752 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:55.756711 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bjn8p" event={"ID":"b78b08a9-11af-446a-abce-551fc42a58e6","Type":"ContainerStarted","Data":"c8a99395dcbfbd7908500ecb56d968e1034a88e58f1d46b8c49654ca06e157bf"} Nov 03 16:12:55.757185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:55.756778 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:12:55.777422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:55.777376 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-bjn8p" podStartSLOduration=1.366364914 podStartE2EDuration="4.777362979s" totalImagesPullingTime="3.410998065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:12:55.776362958 +0000 UTC m=+579.287730827" watchObservedRunningTime="2026-11-03 16:12:55.777362979 +0000 UTC m=+579.288730868" Nov 03 16:12:56.640537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:56.640497 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" path="/var/lib/kubelet/pods/e54799d3-2eee-4ad3-ab1b-931b3dbcfb20/volumes" Nov 03 16:12:57.039020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.038964 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:12:57.039386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.039356 6319 cpu_manager.go:414] "containerMap: removing container" podUID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" containerName="discovery" Nov 03 16:12:57.039386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.039368 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" containerName="discovery" Nov 03 16:12:57.039463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.039452 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54799d3-2eee-4ad3-ab1b-931b3dbcfb20" containerName="discovery" Nov 03 16:12:57.043713 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.043688 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.054349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.054323 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:12:57.074920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.074892 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 16:12:57.124571 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.124535 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtps\" (UniqueName: \"kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps\") pod \"limitador-limitador-74dfcfb894-2zc5r\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.124747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.124726 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file\") pod \"limitador-limitador-74dfcfb894-2zc5r\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.140729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.140696 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:12:57.225945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.225913 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtps\" (UniqueName: \"kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps\") pod \"limitador-limitador-74dfcfb894-2zc5r\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.226164 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.226031 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file\") pod \"limitador-limitador-74dfcfb894-2zc5r\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.226599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.226581 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file\") pod \"limitador-limitador-74dfcfb894-2zc5r\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.239370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.239345 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtps\" (UniqueName: \"kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps\") pod \"limitador-limitador-74dfcfb894-2zc5r\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.361344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.361272 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:12:57.633196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.633173 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:12:57.635715 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:12:57.635679 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f6de1c_0d34_417c_9092_dc62a8b3427b.slice/crio-6512cf9045d9c6dd914e6061d2e0b654962bea0a2ae65a2191414e0a57660eca WatchSource:0}: Error finding container 6512cf9045d9c6dd914e6061d2e0b654962bea0a2ae65a2191414e0a57660eca: Status 404 returned error can't find the container with id 6512cf9045d9c6dd914e6061d2e0b654962bea0a2ae65a2191414e0a57660eca Nov 03 16:12:57.761009 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:12:57.760958 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:12:57.769719 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:12:57.769693 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" event={"ID":"68f6de1c-0d34-417c-9092-dc62a8b3427b","Type":"ContainerStarted","Data":"6512cf9045d9c6dd914e6061d2e0b654962bea0a2ae65a2191414e0a57660eca"} Nov 03 16:13:00.786048 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:00.786007 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" event={"ID":"68f6de1c-0d34-417c-9092-dc62a8b3427b","Type":"ContainerStarted","Data":"d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c"} Nov 03 16:13:00.786622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:00.786213 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:13:00.810651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:00.810607 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" podStartSLOduration=1.0864889070000001 podStartE2EDuration="3.810594436s" totalImagesPullingTime="2.724105529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:00.809821543 +0000 UTC m=+584.321189410" watchObservedRunningTime="2026-11-03 16:13:00.810594436 +0000 UTC m=+584.321962304" Nov 03 16:13:05.761937 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:05.761905 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" Nov 03 16:13:06.767765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:06.767736 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-bjn8p" Nov 03 16:13:11.687084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:11.687045 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:13:11.687508 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:11.687269 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" podUID="68f6de1c-0d34-417c-9092-dc62a8b3427b" containerName="limitador" containerID="cri-o://d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c" gracePeriod=30 Nov 03 16:13:11.687963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:11.687947 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:13:12.236083 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.236059 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:13:12.353364 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.353291 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps\" (UniqueName: \"kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps\") pod \"68f6de1c-0d34-417c-9092-dc62a8b3427b\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " Nov 03 16:13:12.353364 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.353360 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file\" (UniqueName: \"kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file\") pod \"68f6de1c-0d34-417c-9092-dc62a8b3427b\" (UID: \"68f6de1c-0d34-417c-9092-dc62a8b3427b\") " Nov 03 16:13:12.353676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.353657 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file" pod "68f6de1c-0d34-417c-9092-dc62a8b3427b" (UID: "68f6de1c-0d34-417c-9092-dc62a8b3427b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:13:12.355680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.355663 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps" pod "68f6de1c-0d34-417c-9092-dc62a8b3427b" (UID: "68f6de1c-0d34-417c-9092-dc62a8b3427b"). InnerVolumeSpecName "kube-api-access-jhtps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:13:12.454463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.454432 6319 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68f6de1c-0d34-417c-9092-dc62a8b3427b-config-file\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:13:12.454463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.454456 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtps\" (UniqueName: \"kubernetes.io/projected/68f6de1c-0d34-417c-9092-dc62a8b3427b-kube-api-access-jhtps\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:13:12.837524 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.837495 6319 generic.go:356] "Generic (PLEG): container finished" podID="68f6de1c-0d34-417c-9092-dc62a8b3427b" containerID="d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c" exitCode=0 Nov 03 16:13:12.837914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.837535 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" Nov 03 16:13:12.837914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.837581 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" event={"ID":"68f6de1c-0d34-417c-9092-dc62a8b3427b","Type":"ContainerDied","Data":"d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c"} Nov 03 16:13:12.837914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.837637 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2zc5r" event={"ID":"68f6de1c-0d34-417c-9092-dc62a8b3427b","Type":"ContainerDied","Data":"6512cf9045d9c6dd914e6061d2e0b654962bea0a2ae65a2191414e0a57660eca"} Nov 03 16:13:12.837914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.837665 6319 scope.go:122] "RemoveContainer" containerID="d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c" Nov 03 16:13:12.845817 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.845802 6319 scope.go:122] "RemoveContainer" containerID="d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c" Nov 03 16:13:12.846087 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:13:12.846065 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c\": container with ID starting with d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c not found: ID does not exist" containerID="d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c" Nov 03 16:13:12.846148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.846094 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c"} err="failed to get container status \"d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c\": rpc error: code = NotFound desc = could not find container \"d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c\": container with ID starting with d1239ffc79815800aaeb81ff50a4ff71e1cf2fec925e51929a5f0751e1997c8c not found: ID does not exist" Nov 03 16:13:12.854638 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.854621 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:13:12.861482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:12.861457 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2zc5r"] Nov 03 16:13:14.553007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.552950 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7"] Nov 03 16:13:14.553497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.553421 6319 cpu_manager.go:414] "containerMap: removing container" podUID="68f6de1c-0d34-417c-9092-dc62a8b3427b" containerName="limitador" Nov 03 16:13:14.553497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.553440 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f6de1c-0d34-417c-9092-dc62a8b3427b" containerName="limitador" Nov 03 16:13:14.553622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.553540 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f6de1c-0d34-417c-9092-dc62a8b3427b" containerName="limitador" Nov 03 16:13:14.557976 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.557955 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.566434 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.566398 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 16:13:14.566434 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.566408 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:13:14.566581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.566413 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v42rd\"" Nov 03 16:13:14.566581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.566450 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:13:14.566646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.566615 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7"] Nov 03 16:13:14.639962 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.639927 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f6de1c-0d34-417c-9092-dc62a8b3427b" path="/var/lib/kubelet/pods/68f6de1c-0d34-417c-9092-dc62a8b3427b/volumes" Nov 03 16:13:14.674122 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.674093 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fc056e1-e7f2-44d9-8350-bcd37e9e65e7-data\") pod \"postgres-7cdbf6dc44-b7xr7\" (UID: \"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.674258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.674152 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2ch\" (UniqueName: \"kubernetes.io/projected/4fc056e1-e7f2-44d9-8350-bcd37e9e65e7-kube-api-access-ct2ch\") pod \"postgres-7cdbf6dc44-b7xr7\" (UID: \"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.775287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.775255 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2ch\" (UniqueName: \"kubernetes.io/projected/4fc056e1-e7f2-44d9-8350-bcd37e9e65e7-kube-api-access-ct2ch\") pod \"postgres-7cdbf6dc44-b7xr7\" (UID: \"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.775470 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.775333 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fc056e1-e7f2-44d9-8350-bcd37e9e65e7-data\") pod \"postgres-7cdbf6dc44-b7xr7\" (UID: \"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.775721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.775703 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fc056e1-e7f2-44d9-8350-bcd37e9e65e7-data\") pod \"postgres-7cdbf6dc44-b7xr7\" (UID: \"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.788460 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.788440 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2ch\" (UniqueName: \"kubernetes.io/projected/4fc056e1-e7f2-44d9-8350-bcd37e9e65e7-kube-api-access-ct2ch\") pod \"postgres-7cdbf6dc44-b7xr7\" (UID: \"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:14.873247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:14.873165 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:15.146842 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:15.146769 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7"] Nov 03 16:13:15.149729 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:13:15.149702 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc056e1_e7f2_44d9_8350_bcd37e9e65e7.slice/crio-537f58bffeb2fb524c2c9f5cfb330c81da8d1f78b35d32266fd394be21965daf WatchSource:0}: Error finding container 537f58bffeb2fb524c2c9f5cfb330c81da8d1f78b35d32266fd394be21965daf: Status 404 returned error can't find the container with id 537f58bffeb2fb524c2c9f5cfb330c81da8d1f78b35d32266fd394be21965daf Nov 03 16:13:15.851777 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:15.851742 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" event={"ID":"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7","Type":"ContainerStarted","Data":"537f58bffeb2fb524c2c9f5cfb330c81da8d1f78b35d32266fd394be21965daf"} Nov 03 16:13:20.457518 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:20.457496 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 16:13:20.877460 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:20.877428 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" event={"ID":"4fc056e1-e7f2-44d9-8350-bcd37e9e65e7","Type":"ContainerStarted","Data":"a6a8395be74469b0ae8c42a8624fa5cdb68eb9ac23552dcec986cefcac1cfd05"} Nov 03 16:13:20.877662 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:20.877541 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:20.896557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:20.896516 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" podStartSLOduration=1.593717191 podStartE2EDuration="6.896505762s" totalImagesPullingTime="5.302788571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:20.89577171 +0000 UTC m=+604.407139579" watchObservedRunningTime="2026-11-03 16:13:20.896505762 +0000 UTC m=+604.407873620" Nov 03 16:13:25.763465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:25.763438 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ntj89" Nov 03 16:13:26.911755 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:26.911730 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b7xr7" Nov 03 16:13:27.760553 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:13:27.760521 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:13:34.209008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.208962 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgwfv"] Nov 03 16:13:34.223502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.223474 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.229277 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.229248 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgwfv"] Nov 03 16:13:34.230564 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.230540 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 16:13:34.231505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.231483 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 16:13:34.231708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.231689 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 16:13:34.231810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.231705 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7z7w6\"" Nov 03 16:13:34.234959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.234940 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.235071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.234972 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxbt\" (UniqueName: \"kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.235071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.235060 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.335508 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.335477 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.335664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.335540 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.335664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.335580 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxbt\" (UniqueName: \"kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.338081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.338056 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.338189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.338117 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.348516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.348493 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxbt\" (UniqueName: \"kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt\") pod \"maas-controller-579cf7c6d-mgwfv\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.536595 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.536493 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:34.544674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.544653 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4b666674-mnhsn"] Nov 03 16:13:34.547045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.547029 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.575783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.575757 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4b666674-mnhsn"] Nov 03 16:13:34.637239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.637213 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.637394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.637286 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmvv\" (UniqueName: \"kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.637503 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.637479 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.737916 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.737886 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.738084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.737933 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmvv\" (UniqueName: \"kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.738084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.737966 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.740591 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.740563 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.740683 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.740607 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.750539 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.750521 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmvv\" (UniqueName: \"kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv\") pod \"maas-controller-c4b666674-mnhsn\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.817683 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.817609 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgwfv"] Nov 03 16:13:34.820235 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:13:34.820207 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09f94e6_c543_4e79_ae64_d48d79be175c.slice/crio-ac4a7bde1e514766bcbd6fc2c2f9f8bc6dde009cf2d3fa28f1a8fb6c3fb486be WatchSource:0}: Error finding container ac4a7bde1e514766bcbd6fc2c2f9f8bc6dde009cf2d3fa28f1a8fb6c3fb486be: Status 404 returned error can't find the container with id ac4a7bde1e514766bcbd6fc2c2f9f8bc6dde009cf2d3fa28f1a8fb6c3fb486be Nov 03 16:13:34.870441 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.870411 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:34.943972 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:34.941181 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" event={"ID":"e09f94e6-c543-4e79-ae64-d48d79be175c","Type":"ContainerStarted","Data":"ac4a7bde1e514766bcbd6fc2c2f9f8bc6dde009cf2d3fa28f1a8fb6c3fb486be"} Nov 03 16:13:35.158575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:35.158493 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4b666674-mnhsn"] Nov 03 16:13:35.162257 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:13:35.162226 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12a99c4_51ee_4f05_8cf4_e8cee1981bc0.slice/crio-e989a97070de290d77abcac7a1f46d945bdd2c38291b202fe85c0099f2c8fa7d WatchSource:0}: Error finding container e989a97070de290d77abcac7a1f46d945bdd2c38291b202fe85c0099f2c8fa7d: Status 404 returned error can't find the container with id e989a97070de290d77abcac7a1f46d945bdd2c38291b202fe85c0099f2c8fa7d Nov 03 16:13:35.950405 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:35.950368 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-mnhsn" event={"ID":"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0","Type":"ContainerStarted","Data":"e989a97070de290d77abcac7a1f46d945bdd2c38291b202fe85c0099f2c8fa7d"} Nov 03 16:13:37.965745 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:37.965686 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" event={"ID":"e09f94e6-c543-4e79-ae64-d48d79be175c","Type":"ContainerStarted","Data":"40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129"} Nov 03 16:13:37.966205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:37.966003 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:37.999184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:37.999124 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" podStartSLOduration=0.997193887 podStartE2EDuration="3.999088132s" totalImagesPullingTime="3.001894245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:37.996889885 +0000 UTC m=+621.508257754" watchObservedRunningTime="2026-11-03 16:13:37.999088132 +0000 UTC m=+621.510456003" Nov 03 16:13:38.971394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:38.971357 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-mnhsn" event={"ID":"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0","Type":"ContainerStarted","Data":"cb46079233f7679e18b78c1126573d05dcf765af8cef581e3a76df3e50c8507c"} Nov 03 16:13:38.971862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:38.971428 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:38.995423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:38.995342 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-c4b666674-mnhsn" podStartSLOduration=2.070597937 podStartE2EDuration="4.995324016s" totalImagesPullingTime="2.924726079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:38.993391302 +0000 UTC m=+622.504759192" watchObservedRunningTime="2026-11-03 16:13:38.995324016 +0000 UTC m=+622.506691885" Nov 03 16:13:39.634900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.634837 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb"] Nov 03 16:13:39.638472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.638452 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.647781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.647753 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 16:13:39.647939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.647852 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 16:13:39.648219 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.648196 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 16:13:39.648348 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.648285 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-dvrz6\"" Nov 03 16:13:39.650516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.650494 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb"] Nov 03 16:13:39.683609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.683584 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a2dceca0-2a1a-461a-9f2b-7384c3109fca-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.683743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.683643 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6b7x\" (UniqueName: \"kubernetes.io/projected/a2dceca0-2a1a-461a-9f2b-7384c3109fca-kube-api-access-t6b7x\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.683743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.683700 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2dceca0-2a1a-461a-9f2b-7384c3109fca-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.683812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.683740 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dceca0-2a1a-461a-9f2b-7384c3109fca-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.784254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.784219 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a2dceca0-2a1a-461a-9f2b-7384c3109fca-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.784449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.784285 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6b7x\" (UniqueName: \"kubernetes.io/projected/a2dceca0-2a1a-461a-9f2b-7384c3109fca-kube-api-access-t6b7x\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.784449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.784306 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2dceca0-2a1a-461a-9f2b-7384c3109fca-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.784449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.784324 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dceca0-2a1a-461a-9f2b-7384c3109fca-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.784928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.784893 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2dceca0-2a1a-461a-9f2b-7384c3109fca-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.785188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.785166 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a2dceca0-2a1a-461a-9f2b-7384c3109fca-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.786913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.786890 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2dceca0-2a1a-461a-9f2b-7384c3109fca-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.797702 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.797679 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6b7x\" (UniqueName: \"kubernetes.io/projected/a2dceca0-2a1a-461a-9f2b-7384c3109fca-kube-api-access-t6b7x\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4hdlb\" (UID: \"a2dceca0-2a1a-461a-9f2b-7384c3109fca\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:39.953747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:39.953663 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:40.241780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:40.241748 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb"] Nov 03 16:13:40.244812 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:13:40.244783 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dceca0_2a1a_461a_9f2b_7384c3109fca.slice/crio-61939b472f8335e6a77913c3dd6d9c91821b938661da0bbd78a66f2951c10384 WatchSource:0}: Error finding container 61939b472f8335e6a77913c3dd6d9c91821b938661da0bbd78a66f2951c10384: Status 404 returned error can't find the container with id 61939b472f8335e6a77913c3dd6d9c91821b938661da0bbd78a66f2951c10384 Nov 03 16:13:40.981125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:40.981078 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" event={"ID":"a2dceca0-2a1a-461a-9f2b-7384c3109fca","Type":"ContainerStarted","Data":"61939b472f8335e6a77913c3dd6d9c91821b938661da0bbd78a66f2951c10384"} Nov 03 16:13:48.974157 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:48.974120 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:49.978063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:49.978038 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:13:50.032152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.032124 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgwfv"] Nov 03 16:13:50.032340 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.032314 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" podUID="e09f94e6-c543-4e79-ae64-d48d79be175c" containerName="manager" containerID="cri-o://40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129" gracePeriod=30 Nov 03 16:13:50.300220 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.300192 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:50.384755 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.384721 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert\") pod \"e09f94e6-c543-4e79-ae64-d48d79be175c\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " Nov 03 16:13:50.384959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.384767 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs\") pod \"e09f94e6-c543-4e79-ae64-d48d79be175c\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " Nov 03 16:13:50.384959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.384805 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt\" (UniqueName: \"kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt\") pod \"e09f94e6-c543-4e79-ae64-d48d79be175c\" (UID: \"e09f94e6-c543-4e79-ae64-d48d79be175c\") " Nov 03 16:13:50.387472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.387443 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt" pod "e09f94e6-c543-4e79-ae64-d48d79be175c" (UID: "e09f94e6-c543-4e79-ae64-d48d79be175c"). InnerVolumeSpecName "kube-api-access-pjxbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:13:50.387578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.387434 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert" pod "e09f94e6-c543-4e79-ae64-d48d79be175c" (UID: "e09f94e6-c543-4e79-ae64-d48d79be175c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:13:50.387627 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.387571 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs" pod "e09f94e6-c543-4e79-ae64-d48d79be175c" (UID: "e09f94e6-c543-4e79-ae64-d48d79be175c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:13:50.486407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.486368 6319 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-webhook-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:13:50.486407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.486408 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e09f94e6-c543-4e79-ae64-d48d79be175c-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:13:50.486659 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:50.486425 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxbt\" (UniqueName: \"kubernetes.io/projected/e09f94e6-c543-4e79-ae64-d48d79be175c-kube-api-access-pjxbt\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:13:51.036047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.035925 6319 generic.go:356] "Generic (PLEG): container finished" podID="e09f94e6-c543-4e79-ae64-d48d79be175c" containerID="40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129" exitCode=0 Nov 03 16:13:51.036047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.035974 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" Nov 03 16:13:51.036047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.036024 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" event={"ID":"e09f94e6-c543-4e79-ae64-d48d79be175c","Type":"ContainerDied","Data":"40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129"} Nov 03 16:13:51.036663 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.036085 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgwfv" event={"ID":"e09f94e6-c543-4e79-ae64-d48d79be175c","Type":"ContainerDied","Data":"ac4a7bde1e514766bcbd6fc2c2f9f8bc6dde009cf2d3fa28f1a8fb6c3fb486be"} Nov 03 16:13:51.036663 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.036125 6319 scope.go:122] "RemoveContainer" containerID="40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129" Nov 03 16:13:51.046490 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.046474 6319 scope.go:122] "RemoveContainer" containerID="40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129" Nov 03 16:13:51.046807 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:13:51.046784 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129\": container with ID starting with 40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129 not found: ID does not exist" containerID="40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129" Nov 03 16:13:51.046912 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.046812 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129"} err="failed to get container status \"40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129\": rpc error: code = NotFound desc = could not find container \"40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129\": container with ID starting with 40be54b280e75b01ff72fa9aa0ff5ff74226a57ff0c66fb9e962a6bac68b3129 not found: ID does not exist" Nov 03 16:13:51.058270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.058246 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgwfv"] Nov 03 16:13:51.063101 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:51.063076 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgwfv"] Nov 03 16:13:52.641065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:52.641036 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09f94e6-c543-4e79-ae64-d48d79be175c" path="/var/lib/kubelet/pods/e09f94e6-c543-4e79-ae64-d48d79be175c/volumes" Nov 03 16:13:53.046914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:53.046874 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" event={"ID":"a2dceca0-2a1a-461a-9f2b-7384c3109fca","Type":"ContainerStarted","Data":"2c10fcf67acefc2f086b7c30566c4c95fe815621dc56983f17b59c308882e386"} Nov 03 16:13:53.047093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:53.046997 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:13:53.067510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:13:53.067462 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" podStartSLOduration=2.291953382 podStartE2EDuration="14.067449687s" totalImagesPullingTime="11.775496305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:53.067347722 +0000 UTC m=+636.578715590" watchObservedRunningTime="2026-11-03 16:13:53.067449687 +0000 UTC m=+636.578817555" Nov 03 16:13:57.761152 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:13:57.761075 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:14:04.056714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:04.056684 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4hdlb" Nov 03 16:14:09.457013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.456958 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d"] Nov 03 16:14:09.457608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.457312 6319 cpu_manager.go:414] "containerMap: removing container" podUID="e09f94e6-c543-4e79-ae64-d48d79be175c" containerName="manager" Nov 03 16:14:09.457608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.457323 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09f94e6-c543-4e79-ae64-d48d79be175c" containerName="manager" Nov 03 16:14:09.457608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.457405 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09f94e6-c543-4e79-ae64-d48d79be175c" containerName="manager" Nov 03 16:14:09.463353 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.463332 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.473095 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.473065 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 16:14:09.473210 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.473070 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v65q5\"" Nov 03 16:14:09.473287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.473274 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 16:14:09.474862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.474838 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d"] Nov 03 16:14:09.549075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.549034 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.549075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.549076 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.549301 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.549165 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.549455 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.549407 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgh5f\" (UniqueName: \"kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.650354 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.650322 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.650354 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.650354 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.650594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.650370 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.650594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.650409 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgh5f\" (UniqueName: \"kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.651191 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.651167 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.652955 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.652930 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.653137 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.653034 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.663012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.662972 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgh5f\" (UniqueName: \"kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f\") pod \"maas-api-6d669c7fcd-jvz2d\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:09.776911 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:09.776830 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:10.055164 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:10.055028 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d"] Nov 03 16:14:10.057863 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:14:10.057835 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14af299a_3e08_4f02_b1ca_219c4a6827f3.slice/crio-6a877f73ecf6ba129fbe7e22df105492399f6c78b5f1cd63a6c67f4fa3a1c6c5 WatchSource:0}: Error finding container 6a877f73ecf6ba129fbe7e22df105492399f6c78b5f1cd63a6c67f4fa3a1c6c5: Status 404 returned error can't find the container with id 6a877f73ecf6ba129fbe7e22df105492399f6c78b5f1cd63a6c67f4fa3a1c6c5 Nov 03 16:14:10.120928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:10.120900 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" event={"ID":"14af299a-3e08-4f02-b1ca-219c4a6827f3","Type":"ContainerStarted","Data":"6a877f73ecf6ba129fbe7e22df105492399f6c78b5f1cd63a6c67f4fa3a1c6c5"} Nov 03 16:14:13.135315 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:13.135280 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" event={"ID":"14af299a-3e08-4f02-b1ca-219c4a6827f3","Type":"ContainerStarted","Data":"04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f"} Nov 03 16:14:13.135705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:13.135401 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:13.157001 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:13.156929 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" podStartSLOduration=2.014112677 podStartE2EDuration="4.156915436s" totalImagesPullingTime="2.142802759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:13.156235459 +0000 UTC m=+656.667603331" watchObservedRunningTime="2026-11-03 16:14:13.156915436 +0000 UTC m=+656.668283304" Nov 03 16:14:19.144867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:19.144840 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:20.098125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.098094 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q"] Nov 03 16:14:20.118607 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.118574 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q"] Nov 03 16:14:20.118779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.118666 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.145186 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.145156 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.145186 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.145188 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.145553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.145221 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnv6\" (UniqueName: \"kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.145553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.145326 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.246234 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.246199 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnv6\" (UniqueName: \"kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.246234 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.246241 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.246463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.246285 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.246463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.246303 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.247105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.247076 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.248835 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.248817 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.248942 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.248841 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.258865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.258836 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnv6\" (UniqueName: \"kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6\") pod \"maas-api-87dc4db6-9tl8q\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.433536 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.433455 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:20.709131 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:20.709094 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q"] Nov 03 16:14:20.715836 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:14:20.715809 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b72f7e3_78ff_4111_b4c9_a24dfade5ba7.slice/crio-385234d122e0076088ab8868f73c284a0a8baa28d091cd119ce1e8075f6a1a2d WatchSource:0}: Error finding container 385234d122e0076088ab8868f73c284a0a8baa28d091cd119ce1e8075f6a1a2d: Status 404 returned error can't find the container with id 385234d122e0076088ab8868f73c284a0a8baa28d091cd119ce1e8075f6a1a2d Nov 03 16:14:21.168590 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:21.168518 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" event={"ID":"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7","Type":"ContainerStarted","Data":"385234d122e0076088ab8868f73c284a0a8baa28d091cd119ce1e8075f6a1a2d"} Nov 03 16:14:22.174138 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:22.174096 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" event={"ID":"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7","Type":"ContainerStarted","Data":"466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3"} Nov 03 16:14:22.174541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:22.174224 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:22.196152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:22.196097 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" podStartSLOduration=1.733988569 podStartE2EDuration="2.196083609s" totalImagesPullingTime="462.09504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:22.194447751 +0000 UTC m=+665.705815620" watchObservedRunningTime="2026-11-03 16:14:22.196083609 +0000 UTC m=+665.707451477" Nov 03 16:14:27.760819 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:14:27.760784 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:14:28.183471 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.183391 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:14:28.239135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.239105 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d"] Nov 03 16:14:28.239364 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.239341 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" podUID="14af299a-3e08-4f02-b1ca-219c4a6827f3" containerName="maas-api" containerID="cri-o://04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f" gracePeriod=30 Nov 03 16:14:28.486534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.486510 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:28.512488 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.512462 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls\") pod \"14af299a-3e08-4f02-b1ca-219c4a6827f3\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " Nov 03 16:14:28.512636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.512505 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f\" (UniqueName: \"kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f\") pod \"14af299a-3e08-4f02-b1ca-219c4a6827f3\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " Nov 03 16:14:28.512636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.512538 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs\") pod \"14af299a-3e08-4f02-b1ca-219c4a6827f3\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " Nov 03 16:14:28.512636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.512578 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca\") pod \"14af299a-3e08-4f02-b1ca-219c4a6827f3\" (UID: \"14af299a-3e08-4f02-b1ca-219c4a6827f3\") " Nov 03 16:14:28.513661 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.513160 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca" pod "14af299a-3e08-4f02-b1ca-219c4a6827f3" (UID: "14af299a-3e08-4f02-b1ca-219c4a6827f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:14:28.515596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.515537 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls" pod "14af299a-3e08-4f02-b1ca-219c4a6827f3" (UID: "14af299a-3e08-4f02-b1ca-219c4a6827f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:14:28.515695 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.515620 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs" pod "14af299a-3e08-4f02-b1ca-219c4a6827f3" (UID: "14af299a-3e08-4f02-b1ca-219c4a6827f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:14:28.515695 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.515649 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f" pod "14af299a-3e08-4f02-b1ca-219c4a6827f3" (UID: "14af299a-3e08-4f02-b1ca-219c4a6827f3"). InnerVolumeSpecName "kube-api-access-jgh5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:14:28.613757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.613724 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:14:28.613757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.613751 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgh5f\" (UniqueName: \"kubernetes.io/projected/14af299a-3e08-4f02-b1ca-219c4a6827f3-kube-api-access-jgh5f\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:14:28.613757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.613762 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af299a-3e08-4f02-b1ca-219c4a6827f3-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:14:28.614009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:28.613771 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af299a-3e08-4f02-b1ca-219c4a6827f3-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:14:29.203866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.203830 6319 generic.go:356] "Generic (PLEG): container finished" podID="14af299a-3e08-4f02-b1ca-219c4a6827f3" containerID="04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f" exitCode=0 Nov 03 16:14:29.204302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.203880 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" Nov 03 16:14:29.204302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.203928 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" event={"ID":"14af299a-3e08-4f02-b1ca-219c4a6827f3","Type":"ContainerDied","Data":"04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f"} Nov 03 16:14:29.204302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.203998 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d" event={"ID":"14af299a-3e08-4f02-b1ca-219c4a6827f3","Type":"ContainerDied","Data":"6a877f73ecf6ba129fbe7e22df105492399f6c78b5f1cd63a6c67f4fa3a1c6c5"} Nov 03 16:14:29.204302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.204017 6319 scope.go:122] "RemoveContainer" containerID="04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f" Nov 03 16:14:29.212800 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.212783 6319 scope.go:122] "RemoveContainer" containerID="04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f" Nov 03 16:14:29.213060 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:14:29.213040 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f\": container with ID starting with 04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f not found: ID does not exist" containerID="04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f" Nov 03 16:14:29.213105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.213070 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f"} err="failed to get container status \"04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f\": rpc error: code = NotFound desc = could not find container \"04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f\": container with ID starting with 04e4392224c862960c238c159311fd45ea11923237b3fbde9b2bfd6ea818b98f not found: ID does not exist" Nov 03 16:14:29.222716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.222694 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d"] Nov 03 16:14:29.226895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:29.226869 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvz2d"] Nov 03 16:14:30.640825 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:14:30.640783 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14af299a-3e08-4f02-b1ca-219c4a6827f3" path="/var/lib/kubelet/pods/14af299a-3e08-4f02-b1ca-219c4a6827f3/volumes" Nov 03 16:14:57.762172 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:14:57.762136 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:15:14.128506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.128476 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh"] Nov 03 16:15:14.128868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.128836 6319 cpu_manager.go:414] "containerMap: removing container" podUID="14af299a-3e08-4f02-b1ca-219c4a6827f3" containerName="maas-api" Nov 03 16:15:14.128868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.128852 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14af299a-3e08-4f02-b1ca-219c4a6827f3" containerName="maas-api" Nov 03 16:15:14.128942 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.128915 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="14af299a-3e08-4f02-b1ca-219c4a6827f3" containerName="maas-api" Nov 03 16:15:14.132285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.132263 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.141081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.141052 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qbd52\"" Nov 03 16:15:14.141191 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.141081 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 16:15:14.141191 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.141125 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 16:15:14.141281 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.141241 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 16:15:14.148198 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.148179 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh"] Nov 03 16:15:14.196303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.196274 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.196462 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.196309 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.196462 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.196328 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.196462 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.196373 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.196462 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.196427 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84qh\" (UniqueName: \"kubernetes.io/projected/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-kube-api-access-k84qh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.196647 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.196508 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.296893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.296864 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.296911 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297032 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297075 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297100 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297118 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84qh\" (UniqueName: \"kubernetes.io/projected/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-kube-api-access-k84qh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297437 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297415 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297437 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.297510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.297493 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.299841 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.299812 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.299970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.299955 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.309668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.309647 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84qh\" (UniqueName: \"kubernetes.io/projected/c79ffcfe-b586-4ebf-a29e-f2f6c155784c-kube-api-access-k84qh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ggwbh\" (UID: \"c79ffcfe-b586-4ebf-a29e-f2f6c155784c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.445213 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.445130 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:14.735012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:14.734973 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh"] Nov 03 16:15:14.738297 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:14.738274 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79ffcfe_b586_4ebf_a29e_f2f6c155784c.slice/crio-6788dad86beabc965559f65304eb489b3adda12d9ee9ceaa7545dbe54d2f4f5b WatchSource:0}: Error finding container 6788dad86beabc965559f65304eb489b3adda12d9ee9ceaa7545dbe54d2f4f5b: Status 404 returned error can't find the container with id 6788dad86beabc965559f65304eb489b3adda12d9ee9ceaa7545dbe54d2f4f5b Nov 03 16:15:15.397366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.397274 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" event={"ID":"c79ffcfe-b586-4ebf-a29e-f2f6c155784c","Type":"ContainerStarted","Data":"6788dad86beabc965559f65304eb489b3adda12d9ee9ceaa7545dbe54d2f4f5b"} Nov 03 16:15:15.572766 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.570266 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l4d5b"] Nov 03 16:15:15.574952 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.574928 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.580712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.580683 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l4d5b"] Nov 03 16:15:15.583900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.583853 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 16:15:15.607381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.607286 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.607381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.607350 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.608369 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.607468 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcvd\" (UniqueName: \"kubernetes.io/projected/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-kube-api-access-npcvd\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.608369 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.608063 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.608369 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.608111 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.608369 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.608207 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-home\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.709212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.709176 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcvd\" (UniqueName: \"kubernetes.io/projected/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-kube-api-access-npcvd\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.709212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.709212 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.709397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.709257 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.709397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.709297 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-home\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.709397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.709325 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.709497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.709398 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.710425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.710381 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.711519 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.710870 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.711519 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.711140 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-home\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.714127 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.714065 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.714658 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.714615 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.723296 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.723271 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcvd\" (UniqueName: \"kubernetes.io/projected/bed9f35a-eb7a-4d7d-b328-b5fa380c8676-kube-api-access-npcvd\") pod \"echo-simulator-kserve-64b6f8f96c-l4d5b\" (UID: \"bed9f35a-eb7a-4d7d-b328-b5fa380c8676\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:15.898951 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:15.898915 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:16.187973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:16.187939 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l4d5b"] Nov 03 16:15:16.192151 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:16.192109 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed9f35a_eb7a_4d7d_b328_b5fa380c8676.slice/crio-92554a612fdff5157646ade9faa2d9aca146e17c0f17d445e0ee364209ff2c1f WatchSource:0}: Error finding container 92554a612fdff5157646ade9faa2d9aca146e17c0f17d445e0ee364209ff2c1f: Status 404 returned error can't find the container with id 92554a612fdff5157646ade9faa2d9aca146e17c0f17d445e0ee364209ff2c1f Nov 03 16:15:16.403078 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:16.403042 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" event={"ID":"bed9f35a-eb7a-4d7d-b328-b5fa380c8676","Type":"ContainerStarted","Data":"92554a612fdff5157646ade9faa2d9aca146e17c0f17d445e0ee364209ff2c1f"} Nov 03 16:15:16.404400 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:16.404371 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" event={"ID":"c79ffcfe-b586-4ebf-a29e-f2f6c155784c","Type":"ContainerStarted","Data":"da34661827568dd8bd44219baf68b4ae2d4494ebc3ad8193c295f845fe4c3d72"} Nov 03 16:15:16.426422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:16.426380 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" podStartSLOduration=0.904118488 podStartE2EDuration="2.426367919s" totalImagesPullingTime="1.522249431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:16.425091183 +0000 UTC m=+719.936459051" watchObservedRunningTime="2026-11-03 16:15:16.426367919 +0000 UTC m=+719.937735787" Nov 03 16:15:16.445436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:16.445411 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:18.158598 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.158523 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg"] Nov 03 16:15:18.162107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.162086 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.166282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.166254 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 16:15:18.173151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.173123 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg"] Nov 03 16:15:18.230465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.230437 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634e1091-c4df-49da-9b69-037e924c624e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.230642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.230472 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.230642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.230491 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9c9\" (UniqueName: \"kubernetes.io/projected/634e1091-c4df-49da-9b69-037e924c624e-kube-api-access-rn9c9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.230642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.230564 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.230821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.230702 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.230821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.230788 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332102 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332067 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332130 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634e1091-c4df-49da-9b69-037e924c624e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332152 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332169 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9c9\" (UniqueName: \"kubernetes.io/projected/634e1091-c4df-49da-9b69-037e924c624e-kube-api-access-rn9c9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332192 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332216 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332590 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332567 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332590 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332584 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.332672 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.332636 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.334554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.334527 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634e1091-c4df-49da-9b69-037e924c624e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.334837 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.334819 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634e1091-c4df-49da-9b69-037e924c624e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.345352 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.345328 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9c9\" (UniqueName: \"kubernetes.io/projected/634e1091-c4df-49da-9b69-037e924c624e-kube-api-access-rn9c9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg\" (UID: \"634e1091-c4df-49da-9b69-037e924c624e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.415711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.415639 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" event={"ID":"bed9f35a-eb7a-4d7d-b328-b5fa380c8676","Type":"ContainerStarted","Data":"189f1b355f10747357edae57ef99f99c34c94c8d107fa3ea139b7fcb00dc8a8c"} Nov 03 16:15:18.439595 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.439549 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" podStartSLOduration=1.7933497250000001 podStartE2EDuration="3.439536814s" totalImagesPullingTime="1.646187089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:18.439451965 +0000 UTC m=+721.950819832" watchObservedRunningTime="2026-11-03 16:15:18.439536814 +0000 UTC m=+721.950904682" Nov 03 16:15:18.475710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.475684 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:18.752713 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.752680 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg"] Nov 03 16:15:18.755526 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:18.755498 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634e1091_c4df_49da_9b69_037e924c624e.slice/crio-4d67bb5714b080154a5a30e8e302f998b5e3d78334b7b0b237c800961948255f WatchSource:0}: Error finding container 4d67bb5714b080154a5a30e8e302f998b5e3d78334b7b0b237c800961948255f: Status 404 returned error can't find the container with id 4d67bb5714b080154a5a30e8e302f998b5e3d78334b7b0b237c800961948255f Nov 03 16:15:18.899679 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:18.899643 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:19.421501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:19.421414 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" event={"ID":"634e1091-c4df-49da-9b69-037e924c624e","Type":"ContainerStarted","Data":"1685e163430eb16c0bd48ad36567eef5cb344570af5ea60b494b1363040dd239"} Nov 03 16:15:19.422022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:19.421505 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" event={"ID":"634e1091-c4df-49da-9b69-037e924c624e","Type":"ContainerStarted","Data":"4d67bb5714b080154a5a30e8e302f998b5e3d78334b7b0b237c800961948255f"} Nov 03 16:15:19.463820 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:19.463771 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" podStartSLOduration=1.116868595 podStartE2EDuration="1.463755097s" totalImagesPullingTime="346.886502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:19.462739345 +0000 UTC m=+722.974107223" watchObservedRunningTime="2026-11-03 16:15:19.463755097 +0000 UTC m=+722.975122967" Nov 03 16:15:19.476272 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:19.476235 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:22.018684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.018653 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj"] Nov 03 16:15:22.022325 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.022305 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.027203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.027185 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 16:15:22.033044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.032959 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj"] Nov 03 16:15:22.061620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.061584 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.061752 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.061625 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npsm\" (UniqueName: \"kubernetes.io/projected/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-kube-api-access-4npsm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.061752 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.061663 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.061752 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.061735 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.061876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.061783 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.061876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.061812 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163316 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npsm\" (UniqueName: \"kubernetes.io/projected/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-kube-api-access-4npsm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163536 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163373 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163536 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163405 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163536 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163440 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163536 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163464 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163579 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.163957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.163930 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.164092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.164028 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.164092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.164050 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.166110 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.166089 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.166203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.166089 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.175804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.175783 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npsm\" (UniqueName: \"kubernetes.io/projected/cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b-kube-api-access-4npsm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj\" (UID: \"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.339756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.339636 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:22.615956 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:22.615928 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj"] Nov 03 16:15:22.618352 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:22.618325 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb83d6ed_7e19_4b37_b3eb_fe24f40a5f4b.slice/crio-d9a11aec9995f9a8f602696c1ecadd5516317c1b28fbf71e8af085ffa5a13ed9 WatchSource:0}: Error finding container d9a11aec9995f9a8f602696c1ecadd5516317c1b28fbf71e8af085ffa5a13ed9: Status 404 returned error can't find the container with id d9a11aec9995f9a8f602696c1ecadd5516317c1b28fbf71e8af085ffa5a13ed9 Nov 03 16:15:23.439533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:23.439444 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" event={"ID":"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b","Type":"ContainerStarted","Data":"4aa5a3fa79376c112d1c2bc4074eeb88240a6646222e5a1aead6e9edb11cff61"} Nov 03 16:15:23.439533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:23.439501 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" event={"ID":"cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b","Type":"ContainerStarted","Data":"d9a11aec9995f9a8f602696c1ecadd5516317c1b28fbf71e8af085ffa5a13ed9"} Nov 03 16:15:23.462154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:23.462098 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" podStartSLOduration=2.097296283 podStartE2EDuration="2.46208572s" totalImagesPullingTime="364.789437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:23.460600495 +0000 UTC m=+726.971968363" watchObservedRunningTime="2026-11-03 16:15:23.46208572 +0000 UTC m=+726.973453587" Nov 03 16:15:24.340450 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:24.340419 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:24.445973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:24.445936 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:24.458867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:24.458839 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:25.457037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:25.457007 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:15:25.900200 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:25.900116 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:25.912688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:25.912666 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:26.463473 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:26.463445 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:15:27.760998 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:15:27.760942 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:15:28.476812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:28.476779 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:28.489797 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:28.489772 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:29.475904 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:29.475877 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" Nov 03 16:15:32.340459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:32.340401 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:32.352717 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:32.352693 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:32.488044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:32.488020 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj" Nov 03 16:15:36.089715 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.089682 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t"] Nov 03 16:15:36.093897 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.093875 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.109551 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.109525 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t"] Nov 03 16:15:36.179799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.179777 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s22x6\"" Nov 03 16:15:36.190205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190181 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190271 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190222 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190271 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190241 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190292 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190376 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190440 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190413 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190448 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttpp\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190463 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.190556 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.190541 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291447 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291491 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291643 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291623 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291682 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291668 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291692 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291724 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291846 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291747 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291901 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291854 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttpp\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291900 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.291954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.291900 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.292196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.292173 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.292290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.292230 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.292501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.292439 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.292501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.292514 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.294262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.294238 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.294378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.294359 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.304557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.304531 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.304936 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.304912 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttpp\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-tbr8t\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.407969 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.407898 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:36.687146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.687122 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t"] Nov 03 16:15:36.689439 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:36.689412 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddde5f3_5f73_46af_b9d0_220a2c328c00.slice/crio-3ecae862faeb9727ab5e89653873f3b0742ccbe7feadb4e84d9ae601735386f8 WatchSource:0}: Error finding container 3ecae862faeb9727ab5e89653873f3b0742ccbe7feadb4e84d9ae601735386f8: Status 404 returned error can't find the container with id 3ecae862faeb9727ab5e89653873f3b0742ccbe7feadb4e84d9ae601735386f8 Nov 03 16:15:36.691578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:36.691561 6319 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:15:37.501004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:37.500956 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" event={"ID":"fddde5f3-5f73-46af-b9d0-220a2c328c00","Type":"ContainerStarted","Data":"3ecae862faeb9727ab5e89653873f3b0742ccbe7feadb4e84d9ae601735386f8"} Nov 03 16:15:40.733201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:40.733168 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:15:40.733478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:40.733237 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:15:40.733478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:40.733267 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:15:41.541022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:41.540960 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" event={"ID":"fddde5f3-5f73-46af-b9d0-220a2c328c00","Type":"ContainerStarted","Data":"46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1"} Nov 03 16:15:41.568119 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:41.568072 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podStartSLOduration=1.527481729 podStartE2EDuration="5.568059376s" totalImagesPullingTime="4.040577647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:41.566209493 +0000 UTC m=+745.077577397" watchObservedRunningTime="2026-11-03 16:15:41.568059376 +0000 UTC m=+745.079427244" Nov 03 16:15:42.408386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:42.408357 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:42.409573 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:42.409545 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:42.409691 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:42.409599 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:42.998207 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:42.998172 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj"] Nov 03 16:15:43.001767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.001751 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.023544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.022338 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 16:15:43.023544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.022475 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-n6vxs\"" Nov 03 16:15:43.035232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.035202 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 16:15:43.042733 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.042697 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj"] Nov 03 16:15:43.053314 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.053288 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-whsfj\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.053469 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.053337 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zs5\" (UniqueName: \"kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5\") pod \"ai-gateway-controller-7cb8f6bb64-whsfj\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.154711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.154670 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zs5\" (UniqueName: \"kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5\") pod \"ai-gateway-controller-7cb8f6bb64-whsfj\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.154893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.154768 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-whsfj\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.157488 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.157461 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-whsfj\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.169713 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.169681 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zs5\" (UniqueName: \"kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5\") pod \"ai-gateway-controller-7cb8f6bb64-whsfj\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.345082 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.344965 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:43.409402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.408516 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:43.409402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.408600 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:43.639000 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:43.638941 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc491da8f_ba0b_4a3e_bd8b_340360acaf7e.slice/crio-5160b77e97949c92723d3288de21098c16337f654f57822828e3ac3bb00439e6 WatchSource:0}: Error finding container 5160b77e97949c92723d3288de21098c16337f654f57822828e3ac3bb00439e6: Status 404 returned error can't find the container with id 5160b77e97949c92723d3288de21098c16337f654f57822828e3ac3bb00439e6 Nov 03 16:15:43.639143 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:43.639037 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj"] Nov 03 16:15:44.408655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:44.408625 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:44.408827 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:44.408687 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:44.554913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:44.554878 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" event={"ID":"c491da8f-ba0b-4a3e-bd8b-340360acaf7e","Type":"ContainerStarted","Data":"5160b77e97949c92723d3288de21098c16337f654f57822828e3ac3bb00439e6"} Nov 03 16:15:45.409329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:45.409293 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:45.409510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:45.409369 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:46.408798 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:46.408766 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:46.409212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:46.408815 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:46.565996 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:46.565946 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" event={"ID":"c491da8f-ba0b-4a3e-bd8b-340360acaf7e","Type":"ContainerStarted","Data":"f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6"} Nov 03 16:15:46.566163 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:46.566052 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:46.586087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:46.586041 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" podStartSLOduration=2.4474612000000002 podStartE2EDuration="4.586018579s" totalImagesPullingTime="2.138557379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:46.58500984 +0000 UTC m=+750.096377705" watchObservedRunningTime="2026-11-03 16:15:46.586018579 +0000 UTC m=+750.097386472" Nov 03 16:15:47.408862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.408827 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:47.409374 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.408901 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:47.640329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.640297 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-87krw"] Nov 03 16:15:47.643864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.643848 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.655728 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.655691 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 16:15:47.658679 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.658658 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-87krw"] Nov 03 16:15:47.681402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.681341 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g"] Nov 03 16:15:47.685541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.685522 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.693874 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.693849 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g"] Nov 03 16:15:47.694011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.693933 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fkm6f\"" Nov 03 16:15:47.696195 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.696169 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg44s\" (UniqueName: \"kubernetes.io/projected/5179a7ba-833e-4b8b-9ce5-f2b5a77e0253-kube-api-access-fg44s\") pod \"payload-processing-55fbbbf9dc-87krw\" (UID: \"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.696451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.696429 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179a7ba-833e-4b8b-9ce5-f2b5a77e0253-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-87krw\" (UID: \"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.797606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.797566 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg44s\" (UniqueName: \"kubernetes.io/projected/5179a7ba-833e-4b8b-9ce5-f2b5a77e0253-kube-api-access-fg44s\") pod \"payload-processing-55fbbbf9dc-87krw\" (UID: \"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.797801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.797619 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c408781-ebf6-49f4-bab3-5e368dc99aab-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-l4v5g\" (UID: \"7c408781-ebf6-49f4-bab3-5e368dc99aab\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.797801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.797682 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzhv6\" (UniqueName: \"kubernetes.io/projected/7c408781-ebf6-49f4-bab3-5e368dc99aab-kube-api-access-hzhv6\") pod \"payload-pre-processing-7b98bb678c-l4v5g\" (UID: \"7c408781-ebf6-49f4-bab3-5e368dc99aab\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.797921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.797794 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179a7ba-833e-4b8b-9ce5-f2b5a77e0253-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-87krw\" (UID: \"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.798801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.798775 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179a7ba-833e-4b8b-9ce5-f2b5a77e0253-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-87krw\" (UID: \"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.811045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.810972 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg44s\" (UniqueName: \"kubernetes.io/projected/5179a7ba-833e-4b8b-9ce5-f2b5a77e0253-kube-api-access-fg44s\") pod \"payload-processing-55fbbbf9dc-87krw\" (UID: \"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:47.899365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.899329 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c408781-ebf6-49f4-bab3-5e368dc99aab-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-l4v5g\" (UID: \"7c408781-ebf6-49f4-bab3-5e368dc99aab\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.899521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.899395 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzhv6\" (UniqueName: \"kubernetes.io/projected/7c408781-ebf6-49f4-bab3-5e368dc99aab-kube-api-access-hzhv6\") pod \"payload-pre-processing-7b98bb678c-l4v5g\" (UID: \"7c408781-ebf6-49f4-bab3-5e368dc99aab\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.900113 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.900093 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c408781-ebf6-49f4-bab3-5e368dc99aab-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-l4v5g\" (UID: \"7c408781-ebf6-49f4-bab3-5e368dc99aab\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.912144 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.912108 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzhv6\" (UniqueName: \"kubernetes.io/projected/7c408781-ebf6-49f4-bab3-5e368dc99aab-kube-api-access-hzhv6\") pod \"payload-pre-processing-7b98bb678c-l4v5g\" (UID: \"7c408781-ebf6-49f4-bab3-5e368dc99aab\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:47.956889 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:47.956855 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:48.001240 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.001207 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:48.244103 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.244028 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-87krw"] Nov 03 16:15:48.247131 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:48.247097 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5179a7ba_833e_4b8b_9ce5_f2b5a77e0253.slice/crio-0597f5b004cdfa5c06288bedecca8a29b4fada453dcfdb48a631bc120137931e WatchSource:0}: Error finding container 0597f5b004cdfa5c06288bedecca8a29b4fada453dcfdb48a631bc120137931e: Status 404 returned error can't find the container with id 0597f5b004cdfa5c06288bedecca8a29b4fada453dcfdb48a631bc120137931e Nov 03 16:15:48.307217 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.307150 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g"] Nov 03 16:15:48.310163 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:15:48.310123 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c408781_ebf6_49f4_bab3_5e368dc99aab.slice/crio-bc88e01b9ccd0f5681bc7994802af7c2f76bfd246948e0f704e2e24f2de72487 WatchSource:0}: Error finding container bc88e01b9ccd0f5681bc7994802af7c2f76bfd246948e0f704e2e24f2de72487: Status 404 returned error can't find the container with id bc88e01b9ccd0f5681bc7994802af7c2f76bfd246948e0f704e2e24f2de72487 Nov 03 16:15:48.408677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.408640 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:15:48.408823 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.408701 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:15:48.577925 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.577849 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" event={"ID":"7c408781-ebf6-49f4-bab3-5e368dc99aab","Type":"ContainerStarted","Data":"bc88e01b9ccd0f5681bc7994802af7c2f76bfd246948e0f704e2e24f2de72487"} Nov 03 16:15:48.579301 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:48.579272 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" event={"ID":"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253","Type":"ContainerStarted","Data":"0597f5b004cdfa5c06288bedecca8a29b4fada453dcfdb48a631bc120137931e"} Nov 03 16:15:50.133954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:50.133921 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:50.134803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:50.134783 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:50.135657 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:50.135635 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:15:51.129564 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.129535 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt"] Nov 03 16:15:51.135399 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.135360 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.150718 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.150692 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt"] Nov 03 16:15:51.233586 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.233546 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.233754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.233598 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.233754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.233730 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.233878 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.233856 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdpmn\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.233926 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.233908 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.233968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.233941 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.234087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.234071 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.234167 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.234153 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.234227 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.234190 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.335608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335522 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.335608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335596 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.335828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335633 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.335828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335796 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.335943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335861 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.335943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335911 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.335957 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdpmn\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336021 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336048 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336257 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336090 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336257 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336139 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336455 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.336806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336685 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.337006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.336946 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.338898 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.338865 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.339366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.339344 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.350183 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.350157 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.350302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.350157 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdpmn\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-qpwkt\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.452997 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.452936 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:51.596681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.596597 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" event={"ID":"7c408781-ebf6-49f4-bab3-5e368dc99aab","Type":"ContainerStarted","Data":"b3367eaf2a49923adcb48bfec26a1e08540077eb5573a4c5e2501da6d55b5919"} Nov 03 16:15:51.596833 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.596717 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:15:51.598477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.598452 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" event={"ID":"5179a7ba-833e-4b8b-9ce5-f2b5a77e0253","Type":"ContainerStarted","Data":"c389c806776667e779a69807af371da8a3427cfe448cdaafd875977e10139230"} Nov 03 16:15:51.598643 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.598630 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:15:51.639458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.639412 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" podStartSLOduration=1.835364197 podStartE2EDuration="4.639397692s" totalImagesPullingTime="2.804033495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:51.637701678 +0000 UTC m=+755.149069547" watchObservedRunningTime="2026-11-03 16:15:51.639397692 +0000 UTC m=+755.150765561" Nov 03 16:15:51.639592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.639489 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" podStartSLOduration=1.96288883 podStartE2EDuration="4.639484067s" totalImagesPullingTime="2.676595237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:51.619016752 +0000 UTC m=+755.130384742" watchObservedRunningTime="2026-11-03 16:15:51.639484067 +0000 UTC m=+755.150851934" Nov 03 16:15:51.742742 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.742704 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt"] Nov 03 16:15:51.752615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.752573 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:15:51.752714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.752664 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:15:51.752783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:51.752716 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:15:52.604716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:52.604675 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" event={"ID":"95e59cb1-9640-498e-a9b6-912e9f7465f5","Type":"ContainerStarted","Data":"ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587"} Nov 03 16:15:52.605194 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:52.604738 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" event={"ID":"95e59cb1-9640-498e-a9b6-912e9f7465f5","Type":"ContainerStarted","Data":"d6a2790d7c9fad9dc3baefdc6eb58674bfdec18e791d552e2cbbb7eb6d6eb720"} Nov 03 16:15:53.453239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:53.453200 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:15:53.454582 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:53.454554 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 16:15:53.454708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:53.454607 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 16:15:54.453283 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:54.453243 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 16:15:54.453764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:54.453310 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 16:15:55.453385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:55.453341 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 16:15:55.453801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:55.453411 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 16:15:56.453398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:56.453358 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 16:15:56.453799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:56.453423 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 16:15:57.453513 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.453481 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 16:15:57.453960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.453534 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 16:15:57.574215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.574190 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:15:57.595208 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.595159 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podStartSLOduration=6.595145282 podStartE2EDuration="6.595145282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:52.648312176 +0000 UTC m=+756.159680044" watchObservedRunningTime="2026-11-03 16:15:57.595145282 +0000 UTC m=+761.106513153" Nov 03 16:15:57.764969 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:15:57.764933 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:15:57.839541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.839509 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh"] Nov 03 16:15:57.843417 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.843397 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:57.853388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.853353 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh"] Nov 03 16:15:57.905608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.905569 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6793402c-af75-49c1-a697-b0da04a6336f-metrics-certs\") pod \"ai-gateway-controller-8c5c5b46c-bg6dh\" (UID: \"6793402c-af75-49c1-a697-b0da04a6336f\") " pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:57.905787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:57.905643 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4ww\" (UniqueName: \"kubernetes.io/projected/6793402c-af75-49c1-a697-b0da04a6336f-kube-api-access-nt4ww\") pod \"ai-gateway-controller-8c5c5b46c-bg6dh\" (UID: \"6793402c-af75-49c1-a697-b0da04a6336f\") " pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:58.006756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.006722 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4ww\" (UniqueName: \"kubernetes.io/projected/6793402c-af75-49c1-a697-b0da04a6336f-kube-api-access-nt4ww\") pod \"ai-gateway-controller-8c5c5b46c-bg6dh\" (UID: \"6793402c-af75-49c1-a697-b0da04a6336f\") " pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:58.006927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.006821 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6793402c-af75-49c1-a697-b0da04a6336f-metrics-certs\") pod \"ai-gateway-controller-8c5c5b46c-bg6dh\" (UID: \"6793402c-af75-49c1-a697-b0da04a6336f\") " pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:58.009398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.009369 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6793402c-af75-49c1-a697-b0da04a6336f-metrics-certs\") pod \"ai-gateway-controller-8c5c5b46c-bg6dh\" (UID: \"6793402c-af75-49c1-a697-b0da04a6336f\") " pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:58.019793 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.019734 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4ww\" (UniqueName: \"kubernetes.io/projected/6793402c-af75-49c1-a697-b0da04a6336f-kube-api-access-nt4ww\") pod \"ai-gateway-controller-8c5c5b46c-bg6dh\" (UID: \"6793402c-af75-49c1-a697-b0da04a6336f\") " pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:58.157668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.157630 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:58.441613 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.436910 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh"] Nov 03 16:15:58.453763 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.453732 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 16:15:58.454120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.453822 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 16:15:58.658522 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:58.658488 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" event={"ID":"6793402c-af75-49c1-a697-b0da04a6336f","Type":"ContainerStarted","Data":"438709f23c169bffabb764b5ae28d92436d2c8eaf1ffd643436a02ca4b179a42"} Nov 03 16:15:59.666391 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:59.666349 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" event={"ID":"6793402c-af75-49c1-a697-b0da04a6336f","Type":"ContainerStarted","Data":"04cfa3f97ea1e09c5931d236218002c3de52b5ec99c7d1a04ff131ebe85b1a06"} Nov 03 16:15:59.666781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:59.666428 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:15:59.688798 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:15:59.688739 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" podStartSLOduration=2.268381721 podStartE2EDuration="2.688720998s" totalImagesPullingTime="420.339277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:59.685127852 +0000 UTC m=+763.196495721" watchObservedRunningTime="2026-11-03 16:15:59.688720998 +0000 UTC m=+763.200088868" Nov 03 16:16:00.373731 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:00.373702 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:16:00.374126 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:00.374077 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:16:00.375016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:00.374972 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:16:02.607672 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:02.607646 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-87krw" Nov 03 16:16:02.608189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:02.607721 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l4v5g" Nov 03 16:16:06.131531 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.131494 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc"] Nov 03 16:16:06.135388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.135364 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.148892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.148868 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc"] Nov 03 16:16:06.284584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284549 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284746 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284599 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284746 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284678 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284819 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284751 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284819 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284773 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284881 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284845 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284886 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.284914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.284905 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.285042 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.285029 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5k8\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.385899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.385818 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.385899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.385892 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.385918 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.385941 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386068 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386110 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386180 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5k8\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386229 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386279 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386525 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386466 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386652 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386676 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386844 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386692 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.386844 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.386769 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.388552 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.388531 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.388768 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.388747 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.399084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.399062 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.399250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.399233 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5k8\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wrnjc\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.449255 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.449234 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:06.734270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.734247 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc"] Nov 03 16:16:06.737542 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:06.737513 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db585f2_62f8_4472_b652_c1c154ac97cf.slice/crio-c34b0fe9918a2dd0503ee05b0f2a74700d822f14f0aff39aa379a994a6617349 WatchSource:0}: Error finding container c34b0fe9918a2dd0503ee05b0f2a74700d822f14f0aff39aa379a994a6617349: Status 404 returned error can't find the container with id c34b0fe9918a2dd0503ee05b0f2a74700d822f14f0aff39aa379a994a6617349 Nov 03 16:16:06.740104 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.740073 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:06.740205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.740141 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:06.740205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:06.740170 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:07.705977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:07.705948 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" event={"ID":"4db585f2-62f8-4472-b652-c1c154ac97cf","Type":"ContainerStarted","Data":"1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857"} Nov 03 16:16:07.706386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:07.706036 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" event={"ID":"4db585f2-62f8-4472-b652-c1c154ac97cf","Type":"ContainerStarted","Data":"c34b0fe9918a2dd0503ee05b0f2a74700d822f14f0aff39aa379a994a6617349"} Nov 03 16:16:07.741294 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:07.741244 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podStartSLOduration=1.741231012 podStartE2EDuration="1.741231012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:07.740057677 +0000 UTC m=+771.251425545" watchObservedRunningTime="2026-11-03 16:16:07.741231012 +0000 UTC m=+771.252598879" Nov 03 16:16:08.449573 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:08.449535 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:08.450755 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:08.450730 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 16:16:08.450864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:08.450777 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 16:16:09.450141 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:09.450106 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 16:16:09.450541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:09.450163 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 16:16:10.450171 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:10.450142 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 16:16:10.450538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:10.450196 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 16:16:10.673940 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:10.673910 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8c5c5b46c-bg6dh" Nov 03 16:16:10.739363 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:10.737705 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj"] Nov 03 16:16:10.739363 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:10.738143 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" podUID="c491da8f-ba0b-4a3e-bd8b-340360acaf7e" containerName="manager" containerID="cri-o://f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6" gracePeriod=30 Nov 03 16:16:11.001493 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.001432 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:16:11.160632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.160584 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5\" (UniqueName: \"kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5\") pod \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " Nov 03 16:16:11.160839 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.160678 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs\") pod \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\" (UID: \"c491da8f-ba0b-4a3e-bd8b-340360acaf7e\") " Nov 03 16:16:11.162778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.162747 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5" pod "c491da8f-ba0b-4a3e-bd8b-340360acaf7e" (UID: "c491da8f-ba0b-4a3e-bd8b-340360acaf7e"). InnerVolumeSpecName "kube-api-access-z8zs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:16:11.162876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.162856 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs" pod "c491da8f-ba0b-4a3e-bd8b-340360acaf7e" (UID: "c491da8f-ba0b-4a3e-bd8b-340360acaf7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:16:11.262046 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.261960 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zs5\" (UniqueName: \"kubernetes.io/projected/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-kube-api-access-z8zs5\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:16:11.262046 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.262011 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c491da8f-ba0b-4a3e-bd8b-340360acaf7e-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:16:11.449577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.449546 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 16:16:11.449739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.449602 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 16:16:11.751729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.751687 6319 generic.go:356] "Generic (PLEG): container finished" podID="c491da8f-ba0b-4a3e-bd8b-340360acaf7e" containerID="f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6" exitCode=0 Nov 03 16:16:11.752193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.751737 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" Nov 03 16:16:11.752193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.751739 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" event={"ID":"c491da8f-ba0b-4a3e-bd8b-340360acaf7e","Type":"ContainerDied","Data":"f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6"} Nov 03 16:16:11.752193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.751790 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj" event={"ID":"c491da8f-ba0b-4a3e-bd8b-340360acaf7e","Type":"ContainerDied","Data":"5160b77e97949c92723d3288de21098c16337f654f57822828e3ac3bb00439e6"} Nov 03 16:16:11.752193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.751821 6319 scope.go:122] "RemoveContainer" containerID="f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6" Nov 03 16:16:11.761329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.761312 6319 scope.go:122] "RemoveContainer" containerID="f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6" Nov 03 16:16:11.761623 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:16:11.761604 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6\": container with ID starting with f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6 not found: ID does not exist" containerID="f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6" Nov 03 16:16:11.761691 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.761636 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6"} err="failed to get container status \"f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6\": rpc error: code = NotFound desc = could not find container \"f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6\": container with ID starting with f4ec9bf3b738a39f92ceb4d12e631b588127cc12014913a05b15afab095667b6 not found: ID does not exist" Nov 03 16:16:11.774305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.774274 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj"] Nov 03 16:16:11.778482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:11.778459 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-whsfj"] Nov 03 16:16:12.449871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:12.449831 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 16:16:12.450049 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:12.449886 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 16:16:12.642707 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:12.642666 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c491da8f-ba0b-4a3e-bd8b-340360acaf7e" path="/var/lib/kubelet/pods/c491da8f-ba0b-4a3e-bd8b-340360acaf7e/volumes" Nov 03 16:16:13.449932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:13.449897 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 16:16:13.450345 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:13.449950 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 16:16:15.243404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:15.243369 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:15.243861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:15.243682 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:15.244598 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:15.244578 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:16:27.761000 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:16:27.760940 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:16:38.672724 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.672686 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm"] Nov 03 16:16:38.673365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.673349 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c491da8f-ba0b-4a3e-bd8b-340360acaf7e" containerName="manager" Nov 03 16:16:38.673410 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.673368 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c491da8f-ba0b-4a3e-bd8b-340360acaf7e" containerName="manager" Nov 03 16:16:38.673528 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.673516 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c491da8f-ba0b-4a3e-bd8b-340360acaf7e" containerName="manager" Nov 03 16:16:38.677024 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.677002 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.691929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.691905 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4a19cd-openshift-default-dockercfg-h4cvm\"" Nov 03 16:16:38.692751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.692721 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm"] Nov 03 16:16:38.815928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.815897 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.815928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.815932 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816160 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.815961 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816160 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.816141 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816244 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.816205 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816244 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.816228 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816356 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.816339 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9q4x\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.816372 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.816427 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.816395 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.844386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.844359 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g"] Nov 03 16:16:38.847970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.847947 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.857792 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.857768 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7196a-openshift-default-dockercfg-wxfhk\"" Nov 03 16:16:38.863626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.863606 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g"] Nov 03 16:16:38.918211 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918172 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9q4x\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918249 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918284 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918314 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918340 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918376 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918413 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918446 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918490 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwln\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918523 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918560 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918597 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918621 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.918660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918644 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918675 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.918955 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919291 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919419 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919451 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919485 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919528 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919566 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.919824 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.919669 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.921758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.921734 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.921900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.921881 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.930011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.929926 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9q4x\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.931066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.931045 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:38.955440 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.955415 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l"] Nov 03 16:16:38.960158 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.960128 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:38.973066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.973043 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l"] Nov 03 16:16:38.973305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.973285 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-69cedd-openshift-default-dockercfg-5bh57\"" Nov 03 16:16:38.989678 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:38.989622 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:39.020783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.020758 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.020910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.020802 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.020910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.020858 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021040 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwln\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021084 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021182 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021129 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021182 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021163 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021193 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021250 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021292 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021536 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021675 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021555 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021675 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021608 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.021760 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.021732 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.023711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.023691 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.024080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.024059 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.039765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.039735 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwln\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.040760 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.040711 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.122367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122335 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122377 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122610 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122507 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122610 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122554 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5brv\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122691 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122732 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122753 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122857 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122774 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.122900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.122847 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.161296 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.161270 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:39.223639 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223612 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.223772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223663 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.223772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223726 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.223772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223761 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5brv\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.223943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223860 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.223943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223889 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.223943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223918 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.224118 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223946 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.224118 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.223974 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.225407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.225204 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.225710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.225666 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.225815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.225751 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.226117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.226093 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.226251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.226231 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.227087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.227069 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.227190 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.227173 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.249488 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.249464 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.256026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.256002 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5brv\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.276599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.276577 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:39.284297 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.284271 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4"] Nov 03 16:16:39.288559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.288541 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.299261 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.298442 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm"] Nov 03 16:16:39.304765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.304728 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.304848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.304812 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.304890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.304853 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.322916 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.321355 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4"] Nov 03 16:16:39.365677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.365655 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d569fb-openshift-default-dockercfg-wtrd5\"" Nov 03 16:16:39.431029 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.430995 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431183 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431105 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431183 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431151 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431252 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431181 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431252 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431217 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431369 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431353 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8t8\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431404 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431386 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431421 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.431535 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.431518 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.444502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.444473 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g"] Nov 03 16:16:39.448094 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:39.448063 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19cf4b4_48e2_4276_9a4a_f76af7843f29.slice/crio-3ac219395f855d991efdb4b9010ec2e82098cb1a21d56a33b8f54df15392a93c WatchSource:0}: Error finding container 3ac219395f855d991efdb4b9010ec2e82098cb1a21d56a33b8f54df15392a93c: Status 404 returned error can't find the container with id 3ac219395f855d991efdb4b9010ec2e82098cb1a21d56a33b8f54df15392a93c Nov 03 16:16:39.455775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.455738 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.455839 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.455822 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.455880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.455862 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.534361 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.534320 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535089 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535175 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535588 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535653 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535683 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535727 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535750 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535892 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8t8\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.535951 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.536508 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.536830 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.537739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.537194 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.538810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.538132 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.542749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.542690 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.552838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.552640 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.559543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.559047 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8t8\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.562758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.562466 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.604431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.604227 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l"] Nov 03 16:16:39.608570 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:39.608533 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2f2cbd_6d19_4bc4_91e6_bd343bd5ecc1.slice/crio-ff537b99bc0458b7352ee256d2443f27b49768c54adeeb7b72d70ffe2d16a84d WatchSource:0}: Error finding container ff537b99bc0458b7352ee256d2443f27b49768c54adeeb7b72d70ffe2d16a84d: Status 404 returned error can't find the container with id ff537b99bc0458b7352ee256d2443f27b49768c54adeeb7b72d70ffe2d16a84d Nov 03 16:16:39.611945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.611905 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.612060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.612014 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.612122 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.612059 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.634622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.634379 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:39.898099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.897415 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" event={"ID":"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1","Type":"ContainerStarted","Data":"191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e"} Nov 03 16:16:39.898099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.897482 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" event={"ID":"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1","Type":"ContainerStarted","Data":"ff537b99bc0458b7352ee256d2443f27b49768c54adeeb7b72d70ffe2d16a84d"} Nov 03 16:16:39.902674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.901467 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" event={"ID":"c19cf4b4-48e2-4276-9a4a-f76af7843f29","Type":"ContainerStarted","Data":"e8bd12dbbb294d2ec0a9c5b4a8e518fe8cd2746d9cef9f3f23a041a570a2b794"} Nov 03 16:16:39.902674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.901522 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" event={"ID":"c19cf4b4-48e2-4276-9a4a-f76af7843f29","Type":"ContainerStarted","Data":"3ac219395f855d991efdb4b9010ec2e82098cb1a21d56a33b8f54df15392a93c"} Nov 03 16:16:39.913366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.907197 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" event={"ID":"1a3383e3-d49a-4ea2-9f08-2e466fad2c47","Type":"ContainerStarted","Data":"d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f"} Nov 03 16:16:39.913366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.907247 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" event={"ID":"1a3383e3-d49a-4ea2-9f08-2e466fad2c47","Type":"ContainerStarted","Data":"cdcf23ea487f0b3bbfe1cd81135f77083e437c755d899a9dd7552c3a5651e895"} Nov 03 16:16:39.925650 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.925251 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podStartSLOduration=1.925233875 podStartE2EDuration="1.925233875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:39.924207546 +0000 UTC m=+803.435575416" watchObservedRunningTime="2026-11-03 16:16:39.925233875 +0000 UTC m=+803.436601743" Nov 03 16:16:39.938918 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.938890 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4"] Nov 03 16:16:39.942478 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:39.942451 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a0bf2d_0d7c_457c_9be4_b6e08db98c91.slice/crio-0e2013989c264ec8a6ad8889af859b7a6ae6ea499a9e960461a002ecd71ff7c6 WatchSource:0}: Error finding container 0e2013989c264ec8a6ad8889af859b7a6ae6ea499a9e960461a002ecd71ff7c6: Status 404 returned error can't find the container with id 0e2013989c264ec8a6ad8889af859b7a6ae6ea499a9e960461a002ecd71ff7c6 Nov 03 16:16:39.945201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.945083 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.945201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.945150 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.945201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.945185 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:16:39.950671 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.950631 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podStartSLOduration=1.950617244 podStartE2EDuration="1.950617244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:39.949976255 +0000 UTC m=+803.461344135" watchObservedRunningTime="2026-11-03 16:16:39.950617244 +0000 UTC m=+803.461985120" Nov 03 16:16:39.981432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.981385 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podStartSLOduration=1.9813714199999999 podStartE2EDuration="1.98137142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:39.979618998 +0000 UTC m=+803.490986871" watchObservedRunningTime="2026-11-03 16:16:39.98137142 +0000 UTC m=+803.492739287" Nov 03 16:16:39.990814 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:39.990795 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:40.162118 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.162033 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:40.168079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.168050 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:40.277727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.277698 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:40.283868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.283843 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:40.913952 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.913913 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" event={"ID":"11a0bf2d-0d7c-457c-9be4-b6e08db98c91","Type":"ContainerStarted","Data":"179c5c6d9c3937f1455c520a9a9aae620c3abc604691b56e5bf1fc159587c34b"} Nov 03 16:16:40.914481 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.913978 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" event={"ID":"11a0bf2d-0d7c-457c-9be4-b6e08db98c91","Type":"ContainerStarted","Data":"0e2013989c264ec8a6ad8889af859b7a6ae6ea499a9e960461a002ecd71ff7c6"} Nov 03 16:16:40.914481 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.914139 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:40.914822 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.914782 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:40.915389 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.915371 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:16:40.915501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.915473 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:16:40.940278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.940231 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podStartSLOduration=1.940220431 podStartE2EDuration="1.940220431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:40.940067379 +0000 UTC m=+804.451435246" watchObservedRunningTime="2026-11-03 16:16:40.940220431 +0000 UTC m=+804.451588298" Nov 03 16:16:40.997124 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:40.997095 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:41.634964 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:41.634931 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:41.640287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:41.640264 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:41.920661 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:41.920582 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:41.920661 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:41.920634 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:41.921288 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:41.921270 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:16:41.921543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:41.921525 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:16:44.324169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.324116 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct"] Nov 03 16:16:44.328061 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.328039 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.332019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.331999 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-b132e3-serving-cert\"" Nov 03 16:16:44.341545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.341520 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct"] Nov 03 16:16:44.496441 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.496402 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.496631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.496451 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.496631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.496601 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.496741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.496652 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpch\" (UniqueName: \"kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.597612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.597520 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.597612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.597582 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.597819 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.597660 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.597819 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.597714 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpch\" (UniqueName: \"kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.599471 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.599441 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.600897 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.600875 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.601967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.601946 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.635619 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.635587 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpch\" (UniqueName: \"kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch\") pod \"maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.646864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.646841 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:44.936933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:44.936907 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct"] Nov 03 16:16:44.937092 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:44.936904 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9234b6_d71b_48ab_b92f_29cb98ab9352.slice/crio-1f3672a98281322a43760045f3a33d152d09cc59faa4421848b1a1f5762ec35c WatchSource:0}: Error finding container 1f3672a98281322a43760045f3a33d152d09cc59faa4421848b1a1f5762ec35c: Status 404 returned error can't find the container with id 1f3672a98281322a43760045f3a33d152d09cc59faa4421848b1a1f5762ec35c Nov 03 16:16:45.539627 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.539597 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp"] Nov 03 16:16:45.545364 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.545345 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.551190 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.551157 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d7196a\"" Nov 03 16:16:45.566803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.566782 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp"] Nov 03 16:16:45.571317 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.569267 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2"] Nov 03 16:16:45.573438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.573420 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.590538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.590474 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2"] Nov 03 16:16:45.601235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.601209 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d7196a-dockercfg-vws56\"" Nov 03 16:16:45.713601 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.713564 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.713976 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.713955 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4jp\" (UniqueName: \"kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp\") pod \"payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.714222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.714205 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58tv\" (UniqueName: \"kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv\") pod \"payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.714425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.714408 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.827067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.819572 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.827067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.819697 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4jp\" (UniqueName: \"kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp\") pod \"payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.827067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.819773 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58tv\" (UniqueName: \"kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv\") pod \"payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.827067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.819967 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.827067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.820865 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.827067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.821634 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.840035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.839952 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4jp\" (UniqueName: \"kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp\") pod \"payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.841875 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.841853 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58tv\" (UniqueName: \"kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv\") pod \"payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.861398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.861374 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:45.890504 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.890482 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:45.947222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.947191 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" event={"ID":"7e9234b6-d71b-48ab-b92f-29cb98ab9352","Type":"ContainerStarted","Data":"f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d"} Nov 03 16:16:45.947387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.947242 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" event={"ID":"7e9234b6-d71b-48ab-b92f-29cb98ab9352","Type":"ContainerStarted","Data":"1f3672a98281322a43760045f3a33d152d09cc59faa4421848b1a1f5762ec35c"} Nov 03 16:16:45.947387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.947330 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:45.977070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:45.977015 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" podStartSLOduration=1.408094762 podStartE2EDuration="1.976994999s" totalImagesPullingTime="568.900237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:45.974909176 +0000 UTC m=+809.486277058" watchObservedRunningTime="2026-11-03 16:16:45.976994999 +0000 UTC m=+809.488362859" Nov 03 16:16:46.150480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.150459 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp"] Nov 03 16:16:46.152570 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:46.152530 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c01d5a_fc60_4aee_89be_50ecd3abb448.slice/crio-c052215adbc65f69c5a395d9c20f417ec9b565ecf4e22565269c94e3fcb6bf68 WatchSource:0}: Error finding container c052215adbc65f69c5a395d9c20f417ec9b565ecf4e22565269c94e3fcb6bf68: Status 404 returned error can't find the container with id c052215adbc65f69c5a395d9c20f417ec9b565ecf4e22565269c94e3fcb6bf68 Nov 03 16:16:46.189510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.189352 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2"] Nov 03 16:16:46.192784 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:46.192757 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988c231f_9d8e_4873_8e01_8599e5d904d6.slice/crio-f493ea990779128d52f8d401d676584be4dacf75656bfb3f13d4c9bdb1fe0419 WatchSource:0}: Error finding container f493ea990779128d52f8d401d676584be4dacf75656bfb3f13d4c9bdb1fe0419: Status 404 returned error can't find the container with id f493ea990779128d52f8d401d676584be4dacf75656bfb3f13d4c9bdb1fe0419 Nov 03 16:16:46.954070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.954031 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" event={"ID":"38c01d5a-fc60-4aee-89be-50ecd3abb448","Type":"ContainerStarted","Data":"f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245"} Nov 03 16:16:46.954519 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.954110 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:46.954519 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.954137 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" event={"ID":"38c01d5a-fc60-4aee-89be-50ecd3abb448","Type":"ContainerStarted","Data":"c052215adbc65f69c5a395d9c20f417ec9b565ecf4e22565269c94e3fcb6bf68"} Nov 03 16:16:46.955690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.955659 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" event={"ID":"988c231f-9d8e-4873-8e01-8599e5d904d6","Type":"ContainerStarted","Data":"00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296"} Nov 03 16:16:46.955834 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.955702 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" event={"ID":"988c231f-9d8e-4873-8e01-8599e5d904d6","Type":"ContainerStarted","Data":"f493ea990779128d52f8d401d676584be4dacf75656bfb3f13d4c9bdb1fe0419"} Nov 03 16:16:46.955914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.955850 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:46.977571 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:46.977530 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" podStartSLOduration=1.9775161030000001 podStartE2EDuration="1.977516103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:46.976530817 +0000 UTC m=+810.487898710" watchObservedRunningTime="2026-11-03 16:16:46.977516103 +0000 UTC m=+810.488883970" Nov 03 16:16:47.002056 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.002004 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" podStartSLOduration=2.001963374 podStartE2EDuration="2.001963374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:47.000261809 +0000 UTC m=+810.511629700" watchObservedRunningTime="2026-11-03 16:16:47.001963374 +0000 UTC m=+810.513331243" Nov 03 16:16:47.019359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.019337 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v"] Nov 03 16:16:47.022666 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.022652 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.031328 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.031309 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4a19cd-dockercfg-s94w9\"" Nov 03 16:16:47.035424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.035406 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4a19cd\"" Nov 03 16:16:47.043127 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.043103 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v"] Nov 03 16:16:47.146836 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.146794 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpm6\" (UniqueName: \"kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6\") pod \"payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.147006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.146861 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.247748 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.247715 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpm6\" (UniqueName: \"kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6\") pod \"payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.247937 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.247753 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.248809 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.248787 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.260843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.260817 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpm6\" (UniqueName: \"kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6\") pod \"payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.336519 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.336496 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.619935 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.619904 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v"] Nov 03 16:16:47.621512 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:47.621479 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd90afa_e149_404c_9c16_e3c34c463eec.slice/crio-57ba9fe91bca75e2e3224795388f4af47e22060f3b2662cfe0954b327c7c937f WatchSource:0}: Error finding container 57ba9fe91bca75e2e3224795388f4af47e22060f3b2662cfe0954b327c7c937f: Status 404 returned error can't find the container with id 57ba9fe91bca75e2e3224795388f4af47e22060f3b2662cfe0954b327c7c937f Nov 03 16:16:47.962872 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.962837 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" event={"ID":"cdd90afa-e149-404c-9c16-e3c34c463eec","Type":"ContainerStarted","Data":"7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4"} Nov 03 16:16:47.963329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.962896 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" event={"ID":"cdd90afa-e149-404c-9c16-e3c34c463eec","Type":"ContainerStarted","Data":"57ba9fe91bca75e2e3224795388f4af47e22060f3b2662cfe0954b327c7c937f"} Nov 03 16:16:47.963329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.963082 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:16:47.986615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:47.986566 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" podStartSLOduration=1.98655439 podStartE2EDuration="1.98655439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:47.985419269 +0000 UTC m=+811.496787141" watchObservedRunningTime="2026-11-03 16:16:47.98655439 +0000 UTC m=+811.497922257" Nov 03 16:16:48.020043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.020015 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4"] Nov 03 16:16:48.024005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.023971 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.030175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.030149 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d7196a-serving-cert\"" Nov 03 16:16:48.039172 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.039135 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4"] Nov 03 16:16:48.069435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.069370 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.069596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.069530 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhc7\" (UniqueName: \"kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.069688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.069608 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.069743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.069704 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.170574 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.170541 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhc7\" (UniqueName: \"kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.170747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.170584 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.170747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.170640 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.170866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.170753 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.170866 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:16:48.170843 6319 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-serving-cert: secret "maas-api-e2e-worker-w3-d7196a-serving-cert" not found Nov 03 16:16:48.170968 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:16:48.170923 6319 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls podName:def1b0ba-39a7-47ed-9dcd-bc84b2d14410 nodeName:}" failed. No retries permitted until 2026-11-03 16:16:48.670901219 +0000 UTC m=+812.182269083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls") pod "maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" (UID: "def1b0ba-39a7-47ed-9dcd-bc84b2d14410") : secret "maas-api-e2e-worker-w3-d7196a-serving-cert" not found Nov 03 16:16:48.171418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.171400 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.173329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.173307 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.192278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.192256 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhc7\" (UniqueName: \"kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.684051 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.684013 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.686546 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.686520 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:48.945622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:48.945536 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:49.267862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.267833 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4"] Nov 03 16:16:49.270879 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:49.270853 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef1b0ba_39a7_47ed_9dcd_bc84b2d14410.slice/crio-0bfec1e6c12730854d32efe68851132774de1053a958082247d1f1994aa63cd0 WatchSource:0}: Error finding container 0bfec1e6c12730854d32efe68851132774de1053a958082247d1f1994aa63cd0: Status 404 returned error can't find the container with id 0bfec1e6c12730854d32efe68851132774de1053a958082247d1f1994aa63cd0 Nov 03 16:16:49.891299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.891262 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd"] Nov 03 16:16:49.895844 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.895821 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:49.902912 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.902876 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-d569fb-serving-cert\"" Nov 03 16:16:49.910176 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.910152 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd"] Nov 03 16:16:49.987032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.987000 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" event={"ID":"def1b0ba-39a7-47ed-9dcd-bc84b2d14410","Type":"ContainerStarted","Data":"0f979ba2e4010ada07b3e85e833f2d740c58e3e95d56dd7ab428e59e1d55a38f"} Nov 03 16:16:49.987186 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.987058 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" event={"ID":"def1b0ba-39a7-47ed-9dcd-bc84b2d14410","Type":"ContainerStarted","Data":"0bfec1e6c12730854d32efe68851132774de1053a958082247d1f1994aa63cd0"} Nov 03 16:16:49.987186 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:49.987089 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:50.008581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.008549 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.008726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.008586 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.008726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.008614 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2sx\" (UniqueName: \"kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.008799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.008784 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.109910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.109879 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.110104 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.109917 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.110104 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.109951 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2sx\" (UniqueName: \"kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.110251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.110108 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.110640 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.110610 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.112772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.112748 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.112864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.112847 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.137449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.137430 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2sx\" (UniqueName: \"kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx\") pod \"maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.213257 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.213091 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:50.511730 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.511632 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" podStartSLOduration=3.085325859 podStartE2EDuration="3.511612928s" totalImagesPullingTime="426.287069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:50.061687327 +0000 UTC m=+813.573055222" watchObservedRunningTime="2026-11-03 16:16:50.511612928 +0000 UTC m=+814.022980799" Nov 03 16:16:50.520528 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.520438 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd"] Nov 03 16:16:50.528031 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:50.527976 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66617af0_8b2f_4885_9c99_9b89707d3552.slice/crio-c9bb2a97c48201e12814c4bd4b4fda64bfbe30be9fa3157c9d21ebc4f44065ca WatchSource:0}: Error finding container c9bb2a97c48201e12814c4bd4b4fda64bfbe30be9fa3157c9d21ebc4f44065ca: Status 404 returned error can't find the container with id c9bb2a97c48201e12814c4bd4b4fda64bfbe30be9fa3157c9d21ebc4f44065ca Nov 03 16:16:50.994802 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:50.994777 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" event={"ID":"66617af0-8b2f-4885-9c99-9b89707d3552","Type":"ContainerStarted","Data":"c9bb2a97c48201e12814c4bd4b4fda64bfbe30be9fa3157c9d21ebc4f44065ca"} Nov 03 16:16:51.961802 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:51.961776 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:16:52.002573 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.002294 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" event={"ID":"66617af0-8b2f-4885-9c99-9b89707d3552","Type":"ContainerStarted","Data":"78217ca0aa18ec75b1bc0336feee4308ce3c091f45570425c8a0bd1afd5c2a67"} Nov 03 16:16:52.002573 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.002356 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:52.048972 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.048920 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" podStartSLOduration=2.587625894 podStartE2EDuration="3.048905933s" totalImagesPullingTime="461.280039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:52.035180364 +0000 UTC m=+815.546548231" watchObservedRunningTime="2026-11-03 16:16:52.048905933 +0000 UTC m=+815.560273802" Nov 03 16:16:52.216961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.216893 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg"] Nov 03 16:16:52.221933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.221910 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.230316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.230291 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4a19cd-serving-cert\"" Nov 03 16:16:52.242297 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.242224 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg"] Nov 03 16:16:52.248386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.248362 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.248509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.248423 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.248606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.248583 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.248659 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.248643 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wtg\" (UniqueName: \"kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.350213 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.350174 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.350398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.350244 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.350398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.350327 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.350398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.350369 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wtg\" (UniqueName: \"kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.351049 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.350956 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.352909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.352884 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.353284 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.353264 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.363510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.363492 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wtg\" (UniqueName: \"kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg\") pod \"maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.537869 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.537796 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:52.844555 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:16:52.843666 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4cfda1_4aaa_4c6d_ad0b_96b0d05f7bd4.slice/crio-97605590d29de15c667170fcdd7b02b74a45a82be1b7c20f5c9a6b9d45a552b6 WatchSource:0}: Error finding container 97605590d29de15c667170fcdd7b02b74a45a82be1b7c20f5c9a6b9d45a552b6: Status 404 returned error can't find the container with id 97605590d29de15c667170fcdd7b02b74a45a82be1b7c20f5c9a6b9d45a552b6 Nov 03 16:16:52.845586 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:52.845559 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg"] Nov 03 16:16:53.009530 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:53.009494 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" event={"ID":"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4","Type":"ContainerStarted","Data":"97605590d29de15c667170fcdd7b02b74a45a82be1b7c20f5c9a6b9d45a552b6"} Nov 03 16:16:54.041499 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:54.041463 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" event={"ID":"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4","Type":"ContainerStarted","Data":"5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380"} Nov 03 16:16:54.041970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:54.041531 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:16:54.063477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:54.063418 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" podStartSLOduration=1.551947795 podStartE2EDuration="2.063400064s" totalImagesPullingTime="511.452269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:54.062072525 +0000 UTC m=+817.573440395" watchObservedRunningTime="2026-11-03 16:16:54.063400064 +0000 UTC m=+817.574767932" Nov 03 16:16:56.001254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:56.001172 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:16:57.935008 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:16:57.924358 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:16:58.027245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:58.026194 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:16:58.028611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:58.028588 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:16:58.050236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:58.050200 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:16:58.993260 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:16:58.992195 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:17:00.108248 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:00.107200 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:17:09.746032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:09.521367 6319 generic.go:356] "Generic (PLEG): container finished" podID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerID="e8bd12dbbb294d2ec0a9c5b4a8e518fe8cd2746d9cef9f3f23a041a570a2b794" exitCode=0 Nov 03 16:17:09.746032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:09.521442 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" event={"ID":"c19cf4b4-48e2-4276-9a4a-f76af7843f29","Type":"ContainerDied","Data":"e8bd12dbbb294d2ec0a9c5b4a8e518fe8cd2746d9cef9f3f23a041a570a2b794"} Nov 03 16:17:09.746032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:09.521812 6319 scope.go:122] "RemoveContainer" containerID="e8bd12dbbb294d2ec0a9c5b4a8e518fe8cd2746d9cef9f3f23a041a570a2b794" Nov 03 16:17:09.746032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:09.523058 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:09.746032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:09.523127 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:09.746032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:09.523161 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.542203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.542172 6319 generic.go:356] "Generic (PLEG): container finished" podID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerID="179c5c6d9c3937f1455c520a9a9aae620c3abc604691b56e5bf1fc159587c34b" exitCode=0 Nov 03 16:17:10.542380 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.542244 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" event={"ID":"11a0bf2d-0d7c-457c-9be4-b6e08db98c91","Type":"ContainerDied","Data":"179c5c6d9c3937f1455c520a9a9aae620c3abc604691b56e5bf1fc159587c34b"} Nov 03 16:17:10.542712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.542689 6319 scope.go:122] "RemoveContainer" containerID="179c5c6d9c3937f1455c520a9a9aae620c3abc604691b56e5bf1fc159587c34b" Nov 03 16:17:10.544189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.544159 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.544307 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.544249 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.544307 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.544298 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.544497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.544472 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" event={"ID":"c19cf4b4-48e2-4276-9a4a-f76af7843f29","Type":"ContainerStarted","Data":"af1d0ede3f52a84b68763dc1164e04821651f4b65fbd786164c2b61f1a7b78c7"} Nov 03 16:17:10.553028 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.547152 6319 generic.go:356] "Generic (PLEG): container finished" podID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerID="d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f" exitCode=0 Nov 03 16:17:10.553028 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.547295 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" event={"ID":"1a3383e3-d49a-4ea2-9f08-2e466fad2c47","Type":"ContainerDied","Data":"d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f"} Nov 03 16:17:10.553028 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.547766 6319 scope.go:122] "RemoveContainer" containerID="d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f" Nov 03 16:17:10.554540 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.554510 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.554651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.554579 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.554651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.554615 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.555118 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.555083 6319 generic.go:356] "Generic (PLEG): container finished" podID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerID="191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e" exitCode=0 Nov 03 16:17:10.555118 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.555110 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" event={"ID":"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1","Type":"ContainerDied","Data":"191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e"} Nov 03 16:17:10.555467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.555421 6319 scope.go:122] "RemoveContainer" containerID="191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e" Nov 03 16:17:10.558062 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.558028 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.558152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.558108 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.558196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.558154 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:17:10.914274 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.914191 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:17:10.914274 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:10.914259 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:17:11.161841 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.161757 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:17:11.163805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.163460 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:17:11.163805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.163517 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:17:11.577246 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.577213 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" event={"ID":"11a0bf2d-0d7c-457c-9be4-b6e08db98c91","Type":"ContainerStarted","Data":"6616e8c1e0f6148b01bc33f12fcb797bfc85249bb27fc6a5635e5820c1057d42"} Nov 03 16:17:11.587293 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.587260 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" event={"ID":"1a3383e3-d49a-4ea2-9f08-2e466fad2c47","Type":"ContainerStarted","Data":"01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f"} Nov 03 16:17:11.606005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.605891 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" event={"ID":"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1","Type":"ContainerStarted","Data":"d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8"} Nov 03 16:17:11.637394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.637361 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:17:11.660023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.654131 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:11.660023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.654195 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:11.921286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.921189 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:17:11.923097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.923067 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:17:11.996011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.992562 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:17:11.996011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.992689 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:11.996011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:11.992735 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:12.166020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.164467 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:17:12.166020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.164530 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:17:12.279047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.277597 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:17:12.279381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.279355 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:12.279557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.279532 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:12.673066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.664103 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:12.673066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:12.664173 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:13.001011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.000821 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:13.001011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.000879 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:13.186009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.185134 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:17:13.186009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.185217 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:17:13.280064 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.279471 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:13.280064 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.279535 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:13.764010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.762954 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:13.764010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.763055 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:13.993419 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.993381 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:13.993738 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:13.993699 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:14.168525 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.168426 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:17:14.169144 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.169096 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:17:14.277516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.277483 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:14.277822 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.277779 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:14.636007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.635939 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:14.636335 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.636295 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:14.991566 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.991529 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:14.991866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:14.991837 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:15.163012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.162509 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:17:15.163012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.162599 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:17:15.282461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.282365 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:15.283058 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.283026 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:15.643069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.638610 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:15.643069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.638666 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:15.990419 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.990378 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:15.990709 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:15.990678 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:16.174439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:16.174399 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:17:16.174701 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:16.174675 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:17:16.286293 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:16.279503 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:16.286293 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:16.279589 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:16.704533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:16.704004 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:16.704533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:16.704061 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:17.004016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.003502 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:17.004016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.003567 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:17.279067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.278384 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:17.279067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.278449 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:17.306332 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.306295 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:17:17.308473 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.308453 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:17:17.666086 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.664132 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 16:17:17.666086 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.664201 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 16:17:17.990388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.990352 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:17:17.990568 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:17.990408 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:17:18.277431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:18.277346 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:17:18.277431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:18.277406 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 16:17:18.751742 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:18.751712 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:17:18.753699 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:18.753670 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:17:18.995354 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:18.995328 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:17:18.996250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:18.996232 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:17:19.281315 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:19.281283 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:17:19.281624 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:19.281605 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:17:19.282289 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:17:19.282273 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:17:27.762461 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:17:27.762419 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:17:57.763091 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:17:57.763052 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:18:23.639301 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.639268 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9"] Nov 03 16:18:23.643360 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.643336 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.647603 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.647575 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b132e3-openshift-default-dockercfg-d5hc7\"" Nov 03 16:18:23.657702 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.657676 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9"] Nov 03 16:18:23.731993 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.731932 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.731993 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.731974 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732225 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732102 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732225 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732149 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732224 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732254 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732287 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732444 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732413 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.732485 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.732453 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.751202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.751175 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd"] Nov 03 16:18:23.754992 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.754963 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.765665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.765643 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c09b17-openshift-default-dockercfg-2jjh9\"" Nov 03 16:18:23.773194 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.773163 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd"] Nov 03 16:18:23.833413 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833374 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.833413 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833426 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.833705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833447 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.833705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833602 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.833705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833650 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.833829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833709 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.833829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833737 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgb9\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.833829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833773 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.833934 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833833 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.833934 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833859 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.833934 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833913 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.834099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.833945 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.834099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834019 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.834210 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834180 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.834295 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834277 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.834357 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834300 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.834437 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834416 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.834501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834444 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.834556 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834520 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.834607 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834553 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.834607 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834557 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.834945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.834923 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.835309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.835285 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.836505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.836478 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.836680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.836663 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.886136 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.886065 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.892402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.887683 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:23.935402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935361 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935565 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935433 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935565 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935463 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935565 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935509 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935574 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgb9\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935634 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935657 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935858 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935719 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.935858 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935765 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.936021 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.935891 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.936091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.936029 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.936155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.936135 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.936285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.936250 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.936563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.936426 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.938029 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.938007 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.938185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.938158 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.950254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.950236 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgb9\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.951492 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.951469 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:23.957489 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:23.957469 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:24.069181 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.069147 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:24.242486 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.242462 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9"] Nov 03 16:18:24.246849 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:18:24.246802 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1375f130_dfc4_4340_b574_b539dca36dc5.slice/crio-05c21094e866ea8e21eb8c2207596c714882034e9a50a2e92cdc7d68eb184c9d WatchSource:0}: Error finding container 05c21094e866ea8e21eb8c2207596c714882034e9a50a2e92cdc7d68eb184c9d: Status 404 returned error can't find the container with id 05c21094e866ea8e21eb8c2207596c714882034e9a50a2e92cdc7d68eb184c9d Nov 03 16:18:24.249815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.249781 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:24.249899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.249861 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:24.249899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.249893 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:24.352784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.352522 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd"] Nov 03 16:18:24.355239 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:18:24.355214 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ecd076_c57d_4ba5_874c_70c829037b02.slice/crio-e47cda84a9ff26225ba7b05f1fbad54f1863af78aef40e9c3d4eb6966dc8920c WatchSource:0}: Error finding container e47cda84a9ff26225ba7b05f1fbad54f1863af78aef40e9c3d4eb6966dc8920c: Status 404 returned error can't find the container with id e47cda84a9ff26225ba7b05f1fbad54f1863af78aef40e9c3d4eb6966dc8920c Nov 03 16:18:24.355995 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.355896 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" event={"ID":"1375f130-dfc4-4340-b574-b539dca36dc5","Type":"ContainerStarted","Data":"8d36690bee9fae901faa4e7e2dec9319f6dfdd15aaf9d5bbb58d27463cf62275"} Nov 03 16:18:24.355995 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.355957 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" event={"ID":"1375f130-dfc4-4340-b574-b539dca36dc5","Type":"ContainerStarted","Data":"05c21094e866ea8e21eb8c2207596c714882034e9a50a2e92cdc7d68eb184c9d"} Nov 03 16:18:24.358334 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.358303 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:24.358443 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.358362 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:24.358443 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.358406 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:24.385492 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.385360 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podStartSLOduration=1.385343187 podStartE2EDuration="1.385343187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:24.382774076 +0000 UTC m=+907.894141942" watchObservedRunningTime="2026-11-03 16:18:24.385343187 +0000 UTC m=+907.896711056" Nov 03 16:18:24.958498 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:24.958458 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:25.370367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:25.370268 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" event={"ID":"23ecd076-c57d-4ba5-874c-70c829037b02","Type":"ContainerStarted","Data":"e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11"} Nov 03 16:18:25.370367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:25.370329 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" event={"ID":"23ecd076-c57d-4ba5-874c-70c829037b02","Type":"ContainerStarted","Data":"e47cda84a9ff26225ba7b05f1fbad54f1863af78aef40e9c3d4eb6966dc8920c"} Nov 03 16:18:25.398367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:25.398306 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podStartSLOduration=2.398287859 podStartE2EDuration="2.398287859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:25.395696262 +0000 UTC m=+908.907064154" watchObservedRunningTime="2026-11-03 16:18:25.398287859 +0000 UTC m=+908.909655728" Nov 03 16:18:25.971273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:25.971150 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 16:18:25.971917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:25.971884 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 16:18:26.069599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:26.069252 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:26.071285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:26.070938 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:18:26.071285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:26.071018 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:18:26.964013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:26.958175 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 16:18:26.964013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:26.958233 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 16:18:27.072009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:27.071377 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:18:27.072009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:27.071438 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:18:27.784549 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:18:27.784513 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:18:27.962043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:27.961378 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 16:18:27.962043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:27.961441 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 16:18:28.070647 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:28.070546 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:18:28.070647 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:28.070600 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:18:28.964013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:28.963436 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 16:18:28.964013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:28.963500 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 16:18:29.070639 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:29.070601 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:18:29.070785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:29.070655 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:18:29.965068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:29.959116 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 16:18:29.965068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:29.959175 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 16:18:30.069789 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:30.069747 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:18:30.070010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:30.069819 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:18:30.958654 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:30.958615 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 16:18:30.958873 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:30.958684 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 16:18:31.070476 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:31.070440 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:18:31.070868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:31.070517 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:18:32.146710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:32.145185 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:32.146710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:32.146553 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:32.151430 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:32.151405 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:18:32.433036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:32.432873 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:32.434829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:32.434804 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:32.435905 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:32.435875 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:18:39.244922 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.244884 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9"] Nov 03 16:18:39.249798 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.249772 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.267400 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.267372 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9"] Nov 03 16:18:39.307893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.307855 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.311206 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.311170 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.311514 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.311491 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.311707 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.311686 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgw76\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.311883 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.311865 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.312068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.312052 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.312249 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.312234 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.312399 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.312385 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.312550 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.312537 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.330754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.330704 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-afb990-openshift-default-dockercfg-gc64q\"" Nov 03 16:18:39.413875 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.413844 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.413904 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.413927 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.413946 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgw76\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.413969 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414027 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414064 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414391 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414102 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414391 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414147 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414391 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414322 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414492 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414525 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414711 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.414916 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.414895 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.416407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.416389 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.416689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.416672 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.427551 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.427524 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgw76\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.427656 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.427564 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.572252 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.572180 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:39.859607 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.859578 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9"] Nov 03 16:18:39.861462 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:18:39.861384 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb937f4_88f8_40c9_bc40_615c9b81931a.slice/crio-6ba6d2f62a01e634510f7d03ac154a25a620bdc835633e676aef76e51a73cef6 WatchSource:0}: Error finding container 6ba6d2f62a01e634510f7d03ac154a25a620bdc835633e676aef76e51a73cef6: Status 404 returned error can't find the container with id 6ba6d2f62a01e634510f7d03ac154a25a620bdc835633e676aef76e51a73cef6 Nov 03 16:18:39.863810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.863770 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:39.863908 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.863835 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:39.863908 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:39.863867 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:40.498493 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:40.498459 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" event={"ID":"1eb937f4-88f8-40c9-bc40-615c9b81931a","Type":"ContainerStarted","Data":"30f912c888e4c9137fbd52e60908ed20603e0d678ec3955de3a7fc2a5d3e7527"} Nov 03 16:18:40.498949 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:40.498523 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" event={"ID":"1eb937f4-88f8-40c9-bc40-615c9b81931a","Type":"ContainerStarted","Data":"6ba6d2f62a01e634510f7d03ac154a25a620bdc835633e676aef76e51a73cef6"} Nov 03 16:18:40.540500 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:40.540445 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podStartSLOduration=1.540429485 podStartE2EDuration="1.540429485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:40.524502634 +0000 UTC m=+924.035870502" watchObservedRunningTime="2026-11-03 16:18:40.540429485 +0000 UTC m=+924.051797353" Nov 03 16:18:40.573047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:40.573012 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:40.574305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:40.574286 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:18:40.574361 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:40.574337 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 16:18:41.572923 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:41.572889 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:18:41.573333 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:41.572960 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 16:18:42.573349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:42.573308 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:18:42.573708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:42.573375 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 16:18:43.572812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:43.572782 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:18:43.572963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:43.572837 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 16:18:44.572885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:44.572852 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:18:44.573384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:44.572915 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 16:18:45.575390 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:45.575353 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:18:45.575939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:45.575910 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 16:18:46.693721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:46.693684 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:46.694477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:46.694453 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:46.695889 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:18:46.695851 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:18:57.765140 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:18:57.765102 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:19:04.778354 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.778319 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf"] Nov 03 16:19:04.783377 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.783354 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.796006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.795965 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf"] Nov 03 16:19:04.798321 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.798298 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"kube-root-ca.crt\"" Nov 03 16:19:04.798321 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.798317 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"openshift-service-ca.crt\"" Nov 03 16:19:04.799328 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.799308 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"default-dockercfg-dq8hv\"" Nov 03 16:19:04.799432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.799366 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"distinct-w0-d569fb-kserve-self-signed-certs\"" Nov 03 16:19:04.859491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.859456 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.859690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.859503 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.859690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.859577 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.859690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.859633 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.859862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.859784 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhxm\" (UniqueName: \"kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.859919 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.859860 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.968954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.968915 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhxm\" (UniqueName: \"kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.969290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.969268 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.969422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.969408 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.969549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.969536 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.969666 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.969653 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.969813 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.969800 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.976010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.970376 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.982011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.976974 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:04.982296 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.977003 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:05.040098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.986315 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:05.040098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:04.996824 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhxm\" (UniqueName: \"kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:05.049578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:05.049553 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs\") pod \"distinct-w0-d569fb-kserve-fd8ffb455-nplzf\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:05.098850 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:05.098814 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:05.379287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:05.379252 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf"] Nov 03 16:19:05.381141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:05.381117 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdd53ee9_64ea_4c4b_b209_e4e9be14167d.slice/crio-e8fd23302919c375a9fc0bb0423180cb1393d0e5bb258ab5e2ba0f4372daa0eb WatchSource:0}: Error finding container e8fd23302919c375a9fc0bb0423180cb1393d0e5bb258ab5e2ba0f4372daa0eb: Status 404 returned error can't find the container with id e8fd23302919c375a9fc0bb0423180cb1393d0e5bb258ab5e2ba0f4372daa0eb Nov 03 16:19:05.751549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:05.751513 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" event={"ID":"cdd53ee9-64ea-4c4b-b209-e4e9be14167d","Type":"ContainerStarted","Data":"bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68"} Nov 03 16:19:05.751549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:05.751555 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" event={"ID":"cdd53ee9-64ea-4c4b-b209-e4e9be14167d","Type":"ContainerStarted","Data":"e8fd23302919c375a9fc0bb0423180cb1393d0e5bb258ab5e2ba0f4372daa0eb"} Nov 03 16:19:05.776665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:05.776616 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podStartSLOduration=1.776601607 podStartE2EDuration="1.776601607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:05.774269042 +0000 UTC m=+949.285636910" watchObservedRunningTime="2026-11-03 16:19:05.776601607 +0000 UTC m=+949.287969504" Nov 03 16:19:06.099511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:06.099431 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:09.204772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.204738 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl"] Nov 03 16:19:09.209615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.209592 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.223464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.223385 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl"] Nov 03 16:19:09.235933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.235903 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.235966 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236191 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236261 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236241 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236361 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236340 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236408 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236391 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236462 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236427 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236728 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236513 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6qkk\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.236728 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.236584 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.337966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.337929 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.337966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.337971 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6qkk\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338038 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338075 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338123 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338389 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338239 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338389 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338367 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338393 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338461 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338628 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338507 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338628 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338504 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338850 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338803 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.338850 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.338823 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.339045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.339026 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.340563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.340541 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.340655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.340581 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.350671 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.350646 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.350795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.350697 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6qkk\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.526901 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.526872 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:09.620854 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.620820 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt"] Nov 03 16:19:09.625900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.625876 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.637162 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.637129 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt"] Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641376 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641453 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641532 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvr9b\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641564 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641580 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641611 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641633 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641671 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.642034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.641746 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743331 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvr9b\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743377 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743397 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743418 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743448 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743483 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743556 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743593 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743657 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.743969 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.743939 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.744065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.744016 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.744163 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.744066 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.744163 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.744121 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.744323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.744303 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.746132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.746108 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.746318 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.746298 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.755995 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.755953 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.756105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.756013 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvr9b\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:09.839376 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:09.839340 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46270e98_843d_4adc_9950_4bcf08c9f436.slice/crio-03ae4397616a17e40908be55ab24ab75fc5f2c5243c8cdd528a1734b76371b33 WatchSource:0}: Error finding container 03ae4397616a17e40908be55ab24ab75fc5f2c5243c8cdd528a1734b76371b33: Status 404 returned error can't find the container with id 03ae4397616a17e40908be55ab24ab75fc5f2c5243c8cdd528a1734b76371b33 Nov 03 16:19:09.839493 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.839418 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl"] Nov 03 16:19:09.842701 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.842655 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:09.842784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.842750 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:09.842822 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.842797 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:09.957523 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:09.957486 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:10.287524 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.287489 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt"] Nov 03 16:19:10.290024 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:10.289965 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d44abb0_b332_4714_9585_0be6d576d530.slice/crio-b45e668d511fa16d196931e31e6786ef89fcaeb666ec45bb9f5233cb9669d750 WatchSource:0}: Error finding container b45e668d511fa16d196931e31e6786ef89fcaeb666ec45bb9f5233cb9669d750: Status 404 returned error can't find the container with id b45e668d511fa16d196931e31e6786ef89fcaeb666ec45bb9f5233cb9669d750 Nov 03 16:19:10.292737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.292703 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:10.292823 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.292775 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:10.292823 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.292803 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:10.792562 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.792457 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" event={"ID":"46270e98-843d-4adc-9950-4bcf08c9f436","Type":"ContainerStarted","Data":"ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955"} Nov 03 16:19:10.792562 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.792520 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" event={"ID":"46270e98-843d-4adc-9950-4bcf08c9f436","Type":"ContainerStarted","Data":"03ae4397616a17e40908be55ab24ab75fc5f2c5243c8cdd528a1734b76371b33"} Nov 03 16:19:10.794912 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.794885 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" event={"ID":"6d44abb0-b332-4714-9585-0be6d576d530","Type":"ContainerStarted","Data":"e4f9e6c52cdd2dc9e125fba77c77f1ef23a245c94a1b8e997fc27b6a897a1449"} Nov 03 16:19:10.795138 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.795119 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" event={"ID":"6d44abb0-b332-4714-9585-0be6d576d530","Type":"ContainerStarted","Data":"b45e668d511fa16d196931e31e6786ef89fcaeb666ec45bb9f5233cb9669d750"} Nov 03 16:19:10.868042 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.865234 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podStartSLOduration=1.865214725 podStartE2EDuration="1.865214725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:10.82787501 +0000 UTC m=+954.339242879" watchObservedRunningTime="2026-11-03 16:19:10.865214725 +0000 UTC m=+954.376582594" Nov 03 16:19:10.962013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:10.960122 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:11.532398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:11.532363 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:11.540006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:11.534809 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:19:11.540006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:11.534864 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 16:19:11.965034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:11.962719 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:19:11.965034 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:11.962776 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 16:19:12.534010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:12.532411 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:19:12.534010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:12.532472 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 16:19:12.960480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:12.960439 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:19:12.960815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:12.960787 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 16:19:13.528508 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:13.528474 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:19:13.528867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:13.528768 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 16:19:13.959517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:13.959486 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:19:13.960123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:13.960094 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 16:19:14.543010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:14.529331 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:19:14.543010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:14.529402 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 16:19:14.962006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:14.961383 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:19:14.962006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:14.961447 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 16:19:15.099959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.099924 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:15.152367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.152335 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:15.197039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.179821 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podStartSLOduration=6.179797803 podStartE2EDuration="6.179797803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:10.88680039 +0000 UTC m=+954.398168259" watchObservedRunningTime="2026-11-03 16:19:15.179797803 +0000 UTC m=+958.691165673" Nov 03 16:19:15.541464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.541422 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:19:15.541734 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.541705 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 16:19:15.874054 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.873964 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:19:15.966012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.961384 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:19:15.966012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:15.961463 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 16:19:16.527998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:16.527944 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:19:16.528201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:16.528101 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 16:19:16.959280 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:16.958628 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:19:16.959280 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:16.958704 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 16:19:17.532042 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:17.532013 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:17.532546 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:17.532482 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:17.533335 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:17.533317 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:19:18.610667 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.610631 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x"] Nov 03 16:19:18.811655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.811622 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x"] Nov 03 16:19:18.814197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.814173 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:18.869105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.850498 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"default-dockercfg-4mpx9\"" Nov 03 16:19:18.869105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.850603 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"distinct-w4-4a19cd-kserve-self-signed-certs\"" Nov 03 16:19:18.869105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.856590 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"kube-root-ca.crt\"" Nov 03 16:19:18.869105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.856704 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"openshift-service-ca.crt\"" Nov 03 16:19:18.988009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.979349 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:18.988009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.979421 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nllv\" (UniqueName: \"kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:18.988009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.979498 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:18.988009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.979566 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:18.988009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.979608 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:18.988009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:18.979668 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.049009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.024199 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:19:19.147020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.144771 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:19:19.147227 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.112115 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.147482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.147461 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.160006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.112762 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.160240 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.151032 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.177006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.160517 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.177688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.177660 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nllv\" (UniqueName: \"kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.177903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.177885 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.178533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.178511 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.178736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.178652 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.178872 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.158943 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.179189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.179172 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.212059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.212031 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nllv\" (UniqueName: \"kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv\") pod \"distinct-w4-4a19cd-kserve-76c4b5456c-thn2x\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.508010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.507052 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:19.778932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.778450 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:19.779730 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.779706 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:19.806036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:19.801034 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:19:20.050973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:20.050905 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x"] Nov 03 16:19:20.053134 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:20.053106 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b81517c_8ea1_47ba_ac5d_f45983c8df96.slice/crio-82718e1333d969fec8ae34d790ab35be1a7dbf4713e461ba5725c47188b4e45c WatchSource:0}: Error finding container 82718e1333d969fec8ae34d790ab35be1a7dbf4713e461ba5725c47188b4e45c: Status 404 returned error can't find the container with id 82718e1333d969fec8ae34d790ab35be1a7dbf4713e461ba5725c47188b4e45c Nov 03 16:19:20.138634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:20.138608 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" event={"ID":"9b81517c-8ea1-47ba-ac5d-f45983c8df96","Type":"ContainerStarted","Data":"c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64"} Nov 03 16:19:20.138764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:20.138648 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" event={"ID":"9b81517c-8ea1-47ba-ac5d-f45983c8df96","Type":"ContainerStarted","Data":"82718e1333d969fec8ae34d790ab35be1a7dbf4713e461ba5725c47188b4e45c"} Nov 03 16:19:20.160642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:20.160578 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podStartSLOduration=2.160559102 podStartE2EDuration="2.160559102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:20.158157331 +0000 UTC m=+963.669525200" watchObservedRunningTime="2026-11-03 16:19:20.160559102 +0000 UTC m=+963.671926970" Nov 03 16:19:20.507667 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:20.507625 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:24.411099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.411036 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw"] Nov 03 16:19:24.416223 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.416192 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.435600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.435576 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw"] Nov 03 16:19:24.477233 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.477196 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.477468 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.477449 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.477641 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.477624 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.477757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.477743 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.477853 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.477839 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.478066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.478047 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.478206 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.478192 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.478380 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.478366 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.478540 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.478513 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k866v\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.579865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.579820 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.579883 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k866v\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.579926 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.579949 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580272 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580322 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580351 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580453 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580472 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580613 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580649 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580744 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.580971 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.580950 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.581120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.581058 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.582944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.582919 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.583098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.583078 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.598692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.598662 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.598835 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.598705 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k866v\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:24.733098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:24.733064 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:25.020078 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.020051 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw"] Nov 03 16:19:25.022772 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:25.022743 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc432fb68_4436_42fe_86cc_eaddc04711d1.slice/crio-b7deed67f7d7dfc94ea1fb63d4b973006f18728c6c54df25645a12908bc133cd WatchSource:0}: Error finding container b7deed67f7d7dfc94ea1fb63d4b973006f18728c6c54df25645a12908bc133cd: Status 404 returned error can't find the container with id b7deed67f7d7dfc94ea1fb63d4b973006f18728c6c54df25645a12908bc133cd Nov 03 16:19:25.025312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.025274 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:25.025398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.025359 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:25.025453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.025428 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:19:25.170383 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.170343 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" event={"ID":"c432fb68-4436-42fe-86cc-eaddc04711d1","Type":"ContainerStarted","Data":"701bbbc36c3c37f4c094f9f9d5025a6c9c7d3457114f292d95154be14a09b798"} Nov 03 16:19:25.170560 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.170406 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" event={"ID":"c432fb68-4436-42fe-86cc-eaddc04711d1","Type":"ContainerStarted","Data":"b7deed67f7d7dfc94ea1fb63d4b973006f18728c6c54df25645a12908bc133cd"} Nov 03 16:19:25.198300 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.198239 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podStartSLOduration=1.198219585 podStartE2EDuration="1.198219585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:25.19531261 +0000 UTC m=+968.706680479" watchObservedRunningTime="2026-11-03 16:19:25.198219585 +0000 UTC m=+968.709587454" Nov 03 16:19:25.545708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.545028 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5"] Nov 03 16:19:25.550892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.550860 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.562026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.561997 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5"] Nov 03 16:19:25.568370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.567201 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"kube-root-ca.crt\"" Nov 03 16:19:25.568370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.567352 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"openshift-service-ca.crt\"" Nov 03 16:19:25.568370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.567485 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"distinct-w6-afb990-kserve-self-signed-certs\"" Nov 03 16:19:25.568370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.567818 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-afb990\"/\"default-dockercfg-z6l22\"" Nov 03 16:19:25.707013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.705705 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.707013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.705807 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.707013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.705889 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.707013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.705913 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.707013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.705976 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxghl\" (UniqueName: \"kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.707013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.706048 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.736789 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.736758 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:25.807721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.807632 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.807963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.807940 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.808218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.808197 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxghl\" (UniqueName: \"kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.808385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.808369 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.811895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.809303 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.812147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.810710 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.820004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.811949 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.820290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.812333 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.820537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.820517 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.837084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.837026 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.853008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.852811 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxghl\" (UniqueName: \"kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.854355 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.854333 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs\") pod \"distinct-w6-afb990-kserve-78dfb494bd-jrfv5\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:25.865901 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:25.865875 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:26.181743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:26.181715 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5"] Nov 03 16:19:26.183567 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:26.183540 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86b42ef8_935c_4a0d_b01e_848f3bc493bb.slice/crio-1e2b524236a17d2a5f4a9f628f6916af771b976ac4687f41b89992d79326fef0 WatchSource:0}: Error finding container 1e2b524236a17d2a5f4a9f628f6916af771b976ac4687f41b89992d79326fef0: Status 404 returned error can't find the container with id 1e2b524236a17d2a5f4a9f628f6916af771b976ac4687f41b89992d79326fef0 Nov 03 16:19:26.738631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:26.738585 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:19:26.739054 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:26.738662 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 16:19:27.204310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:27.204211 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" event={"ID":"86b42ef8-935c-4a0d-b01e-848f3bc493bb","Type":"ContainerStarted","Data":"409710093bc6da7f4c532f14624bc119aeca75793ee9b63ff6fc8dfac84a4592"} Nov 03 16:19:27.204310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:27.204265 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" event={"ID":"86b42ef8-935c-4a0d-b01e-848f3bc493bb","Type":"ContainerStarted","Data":"1e2b524236a17d2a5f4a9f628f6916af771b976ac4687f41b89992d79326fef0"} Nov 03 16:19:27.239675 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:27.239398 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podStartSLOduration=2.239375536 podStartE2EDuration="2.239375536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:27.22718114 +0000 UTC m=+970.738549009" watchObservedRunningTime="2026-11-03 16:19:27.239375536 +0000 UTC m=+970.750743405" Nov 03 16:19:27.737011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:27.733524 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:19:27.737011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:27.733601 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 16:19:27.765063 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:19:27.765028 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:19:27.866520 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:27.866480 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:28.735023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:28.734435 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:19:28.735023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:28.734504 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 16:19:29.507421 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:29.507381 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:29.544910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:29.544881 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:29.735745 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:29.735707 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:19:29.736079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:29.736044 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 16:19:30.259086 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:30.259054 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:19:30.737010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:30.736091 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:19:30.737010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:30.736160 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 16:19:32.092267 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.092231 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:32.092818 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.092796 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:32.093784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.093763 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:19:32.622710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.622683 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm"] Nov 03 16:19:32.626901 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.626874 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.635947 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.634607 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d569fb\"/\"distinct-2-w0-d569fb-kserve-self-signed-certs\"" Nov 03 16:19:32.638747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.638718 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm"] Nov 03 16:19:32.786825 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.786788 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lqz7\" (UniqueName: \"kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.787032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.786905 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.787032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.786961 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.787032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.787010 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.787160 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.787033 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.787160 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.787147 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.894057 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.889796 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lqz7\" (UniqueName: \"kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.894057 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.889917 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.894057 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.889972 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.894057 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.890027 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.894057 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.890053 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.894057 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.890171 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.914009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.911330 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.919023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.918141 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.919023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.918375 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.928591 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.920058 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.928591 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.927214 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.943371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.939632 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lqz7\" (UniqueName: \"kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7\") pod \"distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:32.949889 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:32.949597 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:33.288796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:33.288770 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm"] Nov 03 16:19:33.290141 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:33.290116 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65133542_1e25_4611_906a_7aded0ed208b.slice/crio-e8738022a852f2fe1b31f55f4ff298f27b6eb625c686a4c4a1f34669dd826ba7 WatchSource:0}: Error finding container e8738022a852f2fe1b31f55f4ff298f27b6eb625c686a4c4a1f34669dd826ba7: Status 404 returned error can't find the container with id e8738022a852f2fe1b31f55f4ff298f27b6eb625c686a4c4a1f34669dd826ba7 Nov 03 16:19:34.273614 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:34.273580 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" event={"ID":"65133542-1e25-4611-906a-7aded0ed208b","Type":"ContainerStarted","Data":"a7fd66e1083e080ef258d55f7c834b61b2868216ebf7476deaeb71aec20c92f6"} Nov 03 16:19:34.273614 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:34.273619 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" event={"ID":"65133542-1e25-4611-906a-7aded0ed208b","Type":"ContainerStarted","Data":"e8738022a852f2fe1b31f55f4ff298f27b6eb625c686a4c4a1f34669dd826ba7"} Nov 03 16:19:34.297803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:34.297751 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podStartSLOduration=2.297736935 podStartE2EDuration="2.297736935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:34.294315502 +0000 UTC m=+977.805683370" watchObservedRunningTime="2026-11-03 16:19:34.297736935 +0000 UTC m=+977.809104803" Nov 03 16:19:34.951396 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:34.951367 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:35.876004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:35.874756 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:36.819104 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:36.797718 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:37.005081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:37.004627 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:19:42.951085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:42.951052 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:42.963756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:42.963733 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:43.473276 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:43.473248 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:19:55.964799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:55.964698 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf"] Nov 03 16:19:55.970962 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:55.970942 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:55.979764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:55.979744 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4a19cd\"/\"distinct-2-w4-4a19cd-kserve-self-signed-certs\"" Nov 03 16:19:55.983244 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:55.983218 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf"] Nov 03 16:19:56.133184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.133143 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.133464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.133445 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.133630 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.133612 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.133789 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.133773 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.133930 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.133915 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.134076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.134056 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w488z\" (UniqueName: \"kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.234948 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.234858 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.234948 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.234916 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.234955 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.235034 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.235069 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w488z\" (UniqueName: \"kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.235158 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235430 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.235402 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235528 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.235473 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.235574 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.235540 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.237543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.237515 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.237728 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.237713 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.247768 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.247744 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w488z\" (UniqueName: \"kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z\") pod \"distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.285509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.285480 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:56.569196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.569171 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf"] Nov 03 16:19:56.570418 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:56.570392 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9214762_f1a2_4155_b8a4_6b01cadc36da.slice/crio-58b956e94264552edceb815a93a140aacb0ffe93fcef252a4942df0323a0a074 WatchSource:0}: Error finding container 58b956e94264552edceb815a93a140aacb0ffe93fcef252a4942df0323a0a074: Status 404 returned error can't find the container with id 58b956e94264552edceb815a93a140aacb0ffe93fcef252a4942df0323a0a074 Nov 03 16:19:56.714454 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.714412 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" event={"ID":"c9214762-f1a2-4155-b8a4-6b01cadc36da","Type":"ContainerStarted","Data":"7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5"} Nov 03 16:19:56.714454 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.714460 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" event={"ID":"c9214762-f1a2-4155-b8a4-6b01cadc36da","Type":"ContainerStarted","Data":"58b956e94264552edceb815a93a140aacb0ffe93fcef252a4942df0323a0a074"} Nov 03 16:19:56.736792 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:56.736742 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podStartSLOduration=1.7367277840000002 podStartE2EDuration="1.736727784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:56.733899261 +0000 UTC m=+1000.245267128" watchObservedRunningTime="2026-11-03 16:19:56.736727784 +0000 UTC m=+1000.248095651" Nov 03 16:19:57.286472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:57.286442 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:19:57.764059 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:19:57.764023 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:19:58.958959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.958926 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8"] Nov 03 16:19:58.963615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.963593 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.967047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.967019 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.967393 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.967363 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfn6\" (UniqueName: \"kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.967544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.967422 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.967544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.967466 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.967676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.967577 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.967676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.967644 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:58.970468 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.970445 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"default-dockercfg-55z92\"" Nov 03 16:19:58.973469 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.973449 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"distinct-w2-b132e3-kserve-self-signed-certs\"" Nov 03 16:19:58.973585 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.973519 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"kube-root-ca.crt\"" Nov 03 16:19:58.975254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.975237 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"openshift-service-ca.crt\"" Nov 03 16:19:58.986372 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:58.986353 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8"] Nov 03 16:19:59.068943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.068907 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.068959 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfn6\" (UniqueName: \"kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069017 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069054 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069131 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069380 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069173 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069436 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069505 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.069593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.069577 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.071459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.071439 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.071725 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.071708 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.082616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.082591 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfn6\" (UniqueName: \"kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6\") pod \"distinct-w2-b132e3-kserve-6f6f49b675-88zc8\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.280388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.280301 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:19:59.562449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.562420 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8"] Nov 03 16:19:59.564024 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:19:59.563978 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd278fa39_ab8f_4010_8e1f_d84e28f83e1f.slice/crio-948a896d754ab165e8b25523fd868b6fe25a38ee5fab84f405567ba9bfafce77 WatchSource:0}: Error finding container 948a896d754ab165e8b25523fd868b6fe25a38ee5fab84f405567ba9bfafce77: Status 404 returned error can't find the container with id 948a896d754ab165e8b25523fd868b6fe25a38ee5fab84f405567ba9bfafce77 Nov 03 16:19:59.734384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.734347 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" event={"ID":"d278fa39-ab8f-4010-8e1f-d84e28f83e1f","Type":"ContainerStarted","Data":"8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92"} Nov 03 16:19:59.734384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.734396 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" event={"ID":"d278fa39-ab8f-4010-8e1f-d84e28f83e1f","Type":"ContainerStarted","Data":"948a896d754ab165e8b25523fd868b6fe25a38ee5fab84f405567ba9bfafce77"} Nov 03 16:19:59.756999 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:19:59.756934 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podStartSLOduration=1.756921554 podStartE2EDuration="1.756921554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:59.753510047 +0000 UTC m=+1003.264877916" watchObservedRunningTime="2026-11-03 16:19:59.756921554 +0000 UTC m=+1003.268289421" Nov 03 16:20:00.280711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:00.280671 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:20:06.285848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:06.285812 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:20:06.298936 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:06.298914 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:20:06.851684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:06.851652 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:20:08.823415 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.823382 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh"] Nov 03 16:20:08.827694 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.827677 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.840695 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.840672 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh"] Nov 03 16:20:08.937908 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.937869 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.937946 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.937971 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.938115 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.938156 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.938184 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.938224 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938304 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.938291 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45q4\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:08.938339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:08.938322 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.039932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.039895 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.039940 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040001 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040030 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040334 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040190 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040334 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040283 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040374 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45q4\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040410 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040425 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040529 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040557 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040744 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040706 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.040936 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.040911 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.041553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.041530 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.042855 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.042835 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.043118 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.043101 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.055948 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.055925 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.056517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.056494 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45q4\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.145015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.144915 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:09.280611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.280573 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:20:09.293433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.293412 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:20:09.320722 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.320687 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q"] Nov 03 16:20:09.325484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.325457 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.341528 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.339131 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q"] Nov 03 16:20:09.438213 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.438187 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh"] Nov 03 16:20:09.440908 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:09.440883 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb899c93_ba2a_4a6f_98ef_5c8cdabefa76.slice/crio-beb44a0f5f7f82f50142e280afefb0159fba18bb6fe670231e3087d844c7d65f WatchSource:0}: Error finding container beb44a0f5f7f82f50142e280afefb0159fba18bb6fe670231e3087d844c7d65f: Status 404 returned error can't find the container with id beb44a0f5f7f82f50142e280afefb0159fba18bb6fe670231e3087d844c7d65f Nov 03 16:20:09.443855 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.443825 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:09.443927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.443910 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:09.443965 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.443945 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:09.445295 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445272 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445318 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445398 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445451 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445538 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445603 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55wvk\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445710 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445686 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445777 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445761 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.445838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.445820 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547032 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55wvk\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547093 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547162 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547193 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547399 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547333 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547399 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547377 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547399 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547442 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547548 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547489 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547748 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547720 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547762 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.547973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.547954 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.548075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.548056 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.548224 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.548208 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.550174 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.550145 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.550503 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.550483 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.590104 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.590073 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.590338 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.590319 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55wvk\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.648521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.648486 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:09.912226 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.912179 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" event={"ID":"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76","Type":"ContainerStarted","Data":"9fdc3327c8ffddb7bc47dcda2167b8ba5978350118fdaa4b8061fd65b05963a1"} Nov 03 16:20:09.912705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.912265 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" event={"ID":"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76","Type":"ContainerStarted","Data":"beb44a0f5f7f82f50142e280afefb0159fba18bb6fe670231e3087d844c7d65f"} Nov 03 16:20:09.925690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.925657 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:20:09.938976 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.938932 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podStartSLOduration=1.938919104 podStartE2EDuration="1.938919104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:09.938162386 +0000 UTC m=+1013.449530277" watchObservedRunningTime="2026-11-03 16:20:09.938919104 +0000 UTC m=+1013.450286973" Nov 03 16:20:09.971371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.971334 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q"] Nov 03 16:20:09.973922 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:09.973892 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54ad8dd_f076_4c92_abe2_6366a3e7d803.slice/crio-dfc5757037ebd7df88ee9b42ae3428c901d25465feee2923f6a10d770ffc5495 WatchSource:0}: Error finding container dfc5757037ebd7df88ee9b42ae3428c901d25465feee2923f6a10d770ffc5495: Status 404 returned error can't find the container with id dfc5757037ebd7df88ee9b42ae3428c901d25465feee2923f6a10d770ffc5495 Nov 03 16:20:09.976750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.976715 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:09.976844 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.976796 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:09.976907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:09.976840 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:10.145997 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:10.145964 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:10.147225 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:10.147202 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:10.147314 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:10.147253 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:10.944372 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:10.942585 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" event={"ID":"c54ad8dd-f076-4c92-abe2-6366a3e7d803","Type":"ContainerStarted","Data":"13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca"} Nov 03 16:20:10.944372 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:10.942641 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" event={"ID":"c54ad8dd-f076-4c92-abe2-6366a3e7d803","Type":"ContainerStarted","Data":"dfc5757037ebd7df88ee9b42ae3428c901d25465feee2923f6a10d770ffc5495"} Nov 03 16:20:11.146009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:11.145426 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:11.146009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:11.145492 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:11.652028 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:11.651377 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:11.656003 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:11.655205 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:11.656003 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:11.655266 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:12.160151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:12.160107 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:12.160763 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:12.160734 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:12.662952 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:12.662918 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:12.663166 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:12.663096 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:13.154015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:13.150451 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:13.154015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:13.150516 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:13.652011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:13.649688 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:13.652011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:13.649773 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:14.183170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:14.160165 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:14.183170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:14.160408 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:14.694159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:14.694124 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:14.694748 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:14.694717 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:15.186780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:15.186517 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:15.186780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:15.186601 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:15.662722 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:15.662674 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:15.663043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:15.663013 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:16.175380 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.175245 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:16.176259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.176209 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:16.557009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.550107 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:16.839065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.838507 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:16.839065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.838567 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:16.878277 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.841581 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:16.878277 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.866135 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:16.878277 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:16.877471 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:17.145863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:17.145719 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:20:17.145863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:17.145783 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 16:20:17.649574 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:17.649535 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:20:17.650113 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:17.649584 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 16:20:18.658451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.655803 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:18.658451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.656678 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:18.663516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.663489 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:20:18.684683 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.684639 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podStartSLOduration=9.684626289 podStartE2EDuration="9.684626289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:10.989917412 +0000 UTC m=+1014.501285305" watchObservedRunningTime="2026-11-03 16:20:18.684626289 +0000 UTC m=+1022.195994157" Nov 03 16:20:18.774378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.774349 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:18.774837 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.774815 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:18.775735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.775717 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:20:18.947959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.947564 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m"] Nov 03 16:20:18.956520 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.956489 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:18.961490 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.961466 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"default-dockercfg-vkbzw\"" Nov 03 16:20:18.966486 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.966458 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m"] Nov 03 16:20:18.967220 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.967194 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"openshift-service-ca.crt\"" Nov 03 16:20:18.967496 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.967473 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"distinct-2-w5-69cedd-kserve-self-signed-certs\"" Nov 03 16:20:18.967678 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:18.967646 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-69cedd\"/\"kube-root-ca.crt\"" Nov 03 16:20:19.000435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.000407 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.000435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.000439 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.000637 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.000577 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.000637 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.000607 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjvk\" (UniqueName: \"kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.000768 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.000745 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.000869 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.000854 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.136821 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.136939 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.137131 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.137172 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.137280 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.137310 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjvk\" (UniqueName: \"kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.138016 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.138674 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.139006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.138899 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.151006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.150329 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.154611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.154554 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjvk\" (UniqueName: \"kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.158532 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.156953 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm\") pod \"distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.280688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.280655 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:19.573134 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:19.573056 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m"] Nov 03 16:20:19.578113 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:19.578065 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4aa1a461_d6d4_49dd_8c8c_c1ab8d647df5.slice/crio-8f056af3fc6394aed0798554d7e1f792ba8599c47e99ab0e2a351ae6f38135ae WatchSource:0}: Error finding container 8f056af3fc6394aed0798554d7e1f792ba8599c47e99ab0e2a351ae6f38135ae: Status 404 returned error can't find the container with id 8f056af3fc6394aed0798554d7e1f792ba8599c47e99ab0e2a351ae6f38135ae Nov 03 16:20:20.299235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:20.299197 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" event={"ID":"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5","Type":"ContainerStarted","Data":"23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b"} Nov 03 16:20:20.299653 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:20.299246 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" event={"ID":"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5","Type":"ContainerStarted","Data":"8f056af3fc6394aed0798554d7e1f792ba8599c47e99ab0e2a351ae6f38135ae"} Nov 03 16:20:20.325004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:20.324941 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podStartSLOduration=2.324925494 podStartE2EDuration="2.324925494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:20.322407749 +0000 UTC m=+1023.833775645" watchObservedRunningTime="2026-11-03 16:20:20.324925494 +0000 UTC m=+1023.836293376" Nov 03 16:20:21.281154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:21.281106 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:23.096243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.096201 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9"] Nov 03 16:20:23.101514 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.101486 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.111147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.111118 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"kube-root-ca.crt\"" Nov 03 16:20:23.112869 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.112848 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9"] Nov 03 16:20:23.113389 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.113363 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"openshift-service-ca.crt\"" Nov 03 16:20:23.125168 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.125144 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"default-dockercfg-xbvrx\"" Nov 03 16:20:23.125286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.125181 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c09b17\"/\"distinct-2-w1-c09b17-kserve-self-signed-certs\"" Nov 03 16:20:23.217116 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.217081 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.217288 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.217123 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.217288 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.217256 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4kh\" (UniqueName: \"kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.217397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.217378 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.217494 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.217478 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.217616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.217597 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.325318 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.325387 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.325442 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4kh\" (UniqueName: \"kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.325515 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.325673 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.325745 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.326460 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.326411 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.333878 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.332479 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.334521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.334455 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.334521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.334469 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.340727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.340663 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.371974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.371898 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4kh\" (UniqueName: \"kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh\") pod \"distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.421838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.421798 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:23.713160 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:23.713131 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9"] Nov 03 16:20:23.714924 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:23.714897 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1605541c_2e9c_42fd_9cd0_1606e0606e83.slice/crio-68ec8f81e40a4ed63a583b3d0976d0f4e3136143d12437d1b35bc3844c6d2f47 WatchSource:0}: Error finding container 68ec8f81e40a4ed63a583b3d0976d0f4e3136143d12437d1b35bc3844c6d2f47: Status 404 returned error can't find the container with id 68ec8f81e40a4ed63a583b3d0976d0f4e3136143d12437d1b35bc3844c6d2f47 Nov 03 16:20:24.351287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:24.351248 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" event={"ID":"1605541c-2e9c-42fd-9cd0-1606e0606e83","Type":"ContainerStarted","Data":"e29d73cbf9757619acc7e162052b77749fbb2019a2e9ecb3bfca1b028214430c"} Nov 03 16:20:24.351740 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:24.351302 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" event={"ID":"1605541c-2e9c-42fd-9cd0-1606e0606e83","Type":"ContainerStarted","Data":"68ec8f81e40a4ed63a583b3d0976d0f4e3136143d12437d1b35bc3844c6d2f47"} Nov 03 16:20:24.375521 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:24.375455 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podStartSLOduration=1.37544204 podStartE2EDuration="1.37544204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:24.37162499 +0000 UTC m=+1027.882992858" watchObservedRunningTime="2026-11-03 16:20:24.37544204 +0000 UTC m=+1027.886809908" Nov 03 16:20:24.422759 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:24.422692 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:27.762862 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:20:27.762826 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:20:29.280848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:29.280807 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:29.294020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:29.293959 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:29.404669 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:29.404643 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:20:33.422134 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:33.422100 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:33.434694 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:33.434666 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:34.433785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:34.433762 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:20:36.216010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.214964 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr"] Nov 03 16:20:36.245007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.232891 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.300009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.279968 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr"] Nov 03 16:20:36.337727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.337674 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8qg\" (UniqueName: \"kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.337927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.337755 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.337927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.337827 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.347308 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.345268 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.366112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.361353 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.366112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.361487 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.366112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.349120 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b132e3\"/\"distinct-2-w2-b132e3-kserve-self-signed-certs\"" Nov 03 16:20:36.481043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.480935 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.481382 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.481360 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8qg\" (UniqueName: \"kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.481526 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.481510 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.481647 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.481634 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.481830 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.481817 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.481926 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.481914 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.482516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.482493 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.483480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.483458 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.484250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.484229 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.546931 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.546894 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.553226 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.553198 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8qg\" (UniqueName: \"kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.579724 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.579686 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs\") pod \"distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:36.681288 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:36.681254 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:37.416360 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:37.414423 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:20:37.553384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:37.549070 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:37.730299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:37.729718 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr"] Nov 03 16:20:37.843763 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:37.841828 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 16:20:37.860155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:37.859072 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:20:38.739511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.739467 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" event={"ID":"5759f2d4-c249-49ec-9b38-f38ebf64abfb","Type":"ContainerStarted","Data":"213b6916177c592d528c9bccdb3e97bd50d321f5d9579203e168fd58fde7646c"} Nov 03 16:20:38.747145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.747102 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" event={"ID":"5759f2d4-c249-49ec-9b38-f38ebf64abfb","Type":"ContainerStarted","Data":"0a0be9f93d235773c9a1ca6fde8e1fab98e776f328b75795552d8699e1f0f00d"} Nov 03 16:20:38.765866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.765800 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podStartSLOduration=2.765780976 podStartE2EDuration="2.765780976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:38.763846306 +0000 UTC m=+1042.275214176" watchObservedRunningTime="2026-11-03 16:20:38.765780976 +0000 UTC m=+1042.277148839" Nov 03 16:20:38.885907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.885836 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x"] Nov 03 16:20:38.889589 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.889574 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.921747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.921713 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922126 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922295 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922279 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922415 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922399 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922526 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922513 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922654 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922792 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.922928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.922914 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxm7\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.923067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.923051 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:38.923711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:38.923686 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x"] Nov 03 16:20:39.024974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.024937 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025060 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025088 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025107 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025122 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025154 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025183 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025213 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxm7\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025251 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025401 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025517 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025605 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.025655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025598 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.026037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.025975 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.027886 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.027861 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.028132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.028105 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.042515 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.042487 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.043637 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.043616 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxm7\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.113556 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.113520 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx"] Nov 03 16:20:39.118324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.118308 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.141247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.141155 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx"] Nov 03 16:20:39.221552 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.221521 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:39.228027 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.227998 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228073 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228129 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228204 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228377 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228361 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228412 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228385 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228447 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228415 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228536 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2xw\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.228588 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.228577 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.262583 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.262547 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh"] Nov 03 16:20:39.269352 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.269325 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.287499 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.287424 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh"] Nov 03 16:20:39.329866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.329824 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.329866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.329862 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxn4d\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330128 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330059 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330177 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330215 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330318 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330299 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330366 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330463 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330498 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330473 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330542 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330625 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2xw\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330654 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330684 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330716 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330754 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330747 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330837 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330870 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.330963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330899 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330923 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.330963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.330939 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.331270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.331253 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.331599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.331574 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.333951 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.333924 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.334270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.334251 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.346619 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.346574 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.346875 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.346811 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2xw\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.432555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432468 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.432555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432517 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.432771 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432581 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.432771 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432624 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.432771 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432640 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:39.432771 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432652 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxn4d\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432791 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432823 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.432999 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.433028 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433295 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.433098 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433295 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.433227 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.433406 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.433493 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.433970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.433948 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.435605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.435577 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.435725 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.435709 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.446218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.446196 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.446830 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.446629 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxn4d\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d\") pod \"e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.522501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.522469 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x"] Nov 03 16:20:39.522858 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:39.522837 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cc12de_8730_4db9_a515_5ecd5d85d28f.slice/crio-17ee0098678e08d41ae6e9120a0f6ff6d38736906ca897627fbc5effca6c5d0d WatchSource:0}: Error finding container 17ee0098678e08d41ae6e9120a0f6ff6d38736906ca897627fbc5effca6c5d0d: Status 404 returned error can't find the container with id 17ee0098678e08d41ae6e9120a0f6ff6d38736906ca897627fbc5effca6c5d0d Nov 03 16:20:39.526273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.526241 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.526350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.526305 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.526350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.526335 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.605019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.604965 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:39.683222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.683154 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:39.734587 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.734536 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx"] Nov 03 16:20:39.741846 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.741802 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.742286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.741892 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.742286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.741940 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.751317 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.751286 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" event={"ID":"94cc12de-8730-4db9-a515-5ecd5d85d28f","Type":"ContainerStarted","Data":"eb081d5e3b1dd92edf25f114e4a21ca7f4c90f6be3ce12491c3bee3b49231965"} Nov 03 16:20:39.751428 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.751355 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" event={"ID":"94cc12de-8730-4db9-a515-5ecd5d85d28f","Type":"ContainerStarted","Data":"17ee0098678e08d41ae6e9120a0f6ff6d38736906ca897627fbc5effca6c5d0d"} Nov 03 16:20:39.752957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.752932 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" event={"ID":"7cac05b9-289b-4776-a5b1-768140d2d0a8","Type":"ContainerStarted","Data":"0f89b3698d280e884a842247674fffb4b8a51325e4a27e22f7edeb373ffaf008"} Nov 03 16:20:39.780605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.780555 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podStartSLOduration=1.7805412729999999 podStartE2EDuration="1.780541273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:39.777930878 +0000 UTC m=+1043.289298762" watchObservedRunningTime="2026-11-03 16:20:39.780541273 +0000 UTC m=+1043.291909141" Nov 03 16:20:39.903109 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:39.903074 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc8f7a_b111_43ba_ae49_cc7698dfe3c2.slice/crio-a6452c937e5aee62e6f238b6de54b8fc01ced4b6ab004a4a9d5248606a0182bc WatchSource:0}: Error finding container a6452c937e5aee62e6f238b6de54b8fc01ced4b6ab004a4a9d5248606a0182bc: Status 404 returned error can't find the container with id a6452c937e5aee62e6f238b6de54b8fc01ced4b6ab004a4a9d5248606a0182bc Nov 03 16:20:39.903566 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.903524 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh"] Nov 03 16:20:39.906665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.906594 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.906785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.906672 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:39.906785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:39.906719 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:40.221895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.221862 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:40.223017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.222969 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:40.223127 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.223043 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:40.782162 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.782084 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" event={"ID":"7cac05b9-289b-4776-a5b1-768140d2d0a8","Type":"ContainerStarted","Data":"c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa"} Nov 03 16:20:40.785112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.785072 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" event={"ID":"effc8f7a-b111-43ba-ae49-cc7698dfe3c2","Type":"ContainerStarted","Data":"7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29"} Nov 03 16:20:40.785292 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.785141 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" event={"ID":"effc8f7a-b111-43ba-ae49-cc7698dfe3c2","Type":"ContainerStarted","Data":"a6452c937e5aee62e6f238b6de54b8fc01ced4b6ab004a4a9d5248606a0182bc"} Nov 03 16:20:40.886319 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.886257 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podStartSLOduration=1.886236094 podStartE2EDuration="1.886236094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:40.831435641 +0000 UTC m=+1044.342803510" watchObservedRunningTime="2026-11-03 16:20:40.886236094 +0000 UTC m=+1044.397603964" Nov 03 16:20:40.889007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:40.888762 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podStartSLOduration=1.8887453189999999 podStartE2EDuration="1.888745319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:40.868831484 +0000 UTC m=+1044.380199353" watchObservedRunningTime="2026-11-03 16:20:40.888745319 +0000 UTC m=+1044.400113191" Nov 03 16:20:41.227444 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.227408 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:41.227737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.227713 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:41.475392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.463095 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:41.496677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.496558 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:41.496953 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.496912 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:41.605475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.605443 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:41.606794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.606772 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:41.606955 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:41.606936 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:42.225423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:42.225387 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:42.226132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:42.226098 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:42.506038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:42.505923 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:42.506293 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:42.506254 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:42.607859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:42.606353 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:42.607859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:42.606416 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:43.229010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:43.221912 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:43.229010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:43.221974 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:43.437010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:43.436202 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:43.437010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:43.436265 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:43.605388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:43.605301 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:43.605388 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:43.605365 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:44.234010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:44.224374 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:44.234010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:44.224433 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:44.452010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:44.451097 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:44.452010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:44.451157 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:44.642290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:44.642199 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:44.642551 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:44.642518 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:45.237008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:45.234454 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:45.237008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:45.234519 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:45.452010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:45.451146 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:45.452010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:45.451202 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:45.606091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:45.606006 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:45.606091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:45.606064 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:46.230109 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.226027 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:20:46.230109 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.226088 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 16:20:46.441044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.440592 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:46.441044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.440652 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:46.606050 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.605821 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:46.606050 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.605909 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:46.682445 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.682407 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:46.697133 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.697105 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:46.928131 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:46.928032 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:20:47.434239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.434200 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:20:47.434529 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.434486 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 16:20:47.605809 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.605437 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:20:47.605809 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.605491 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 16:20:47.687308 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.687213 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:47.687660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.687641 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:47.689373 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:47.689348 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:20:48.438665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:48.438586 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:48.439045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:48.439015 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:48.439944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:48.439925 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:20:48.652467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:48.652436 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:48.653247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:48.653226 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:48.654396 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:48.654372 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:20:54.004817 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.004788 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp"] Nov 03 16:20:54.010874 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.010853 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.024735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.024713 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp"] Nov 03 16:20:54.163623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.163583 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx968\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.163813 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.163701 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.163813 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.163754 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.163954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.163835 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.163954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.163904 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.164093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.164035 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.164093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.164080 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.164227 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.164191 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.164289 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.164256 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.267645 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267545 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.267806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267657 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.267806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267694 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.267806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267789 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267843 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267887 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx968\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267944 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267951 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.267976 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.268038 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.268135 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.268328 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268423 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.268333 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.268510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.268494 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.270216 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.270187 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.270597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.270579 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.281697 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.281654 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx968\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.283889 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.283864 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.327634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.327601 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:54.531462 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.531364 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff"] Nov 03 16:20:54.536867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.536846 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.575112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.575074 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff"] Nov 03 16:20:54.613607 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.613568 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p"] Nov 03 16:20:54.618040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.618019 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.651564 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.651534 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p"] Nov 03 16:20:54.655390 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.655363 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp"] Nov 03 16:20:54.660724 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:54.660696 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f71d83_d648_4a63_9289_60904622b040.slice/crio-4b15ce8fc163ae70cc0dd94479727bfaed194ad420ab07afbf36a5882e3bdf60 WatchSource:0}: Error finding container 4b15ce8fc163ae70cc0dd94479727bfaed194ad420ab07afbf36a5882e3bdf60: Status 404 returned error can't find the container with id 4b15ce8fc163ae70cc0dd94479727bfaed194ad420ab07afbf36a5882e3bdf60 Nov 03 16:20:54.668876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.668841 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:54.669037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.668923 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:54.669037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.668953 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680033 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680133 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ztv\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680213 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680275 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680300 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680350 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680454 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.682784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.680499 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.683527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.683499 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785150 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785055 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785150 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785099 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.785150 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785142 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785159 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785180 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.785594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785463 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785549 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.785594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785590 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785717 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785620 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.785847 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.785823 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.787318 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787292 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.787442 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787412 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787616 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787663 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787718 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2jk\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787756 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787926 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.787950 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.788243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.788238 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.788655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.788325 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.788655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.788386 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.788655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.788419 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.788655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.788427 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ztv\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.790662 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.790637 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.790808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.790778 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.798038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.797974 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.801850 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.801822 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ztv\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv\") pod \"e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.853371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.853329 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:54.889857 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.889826 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.889928 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.889978 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890018 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890060 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890109 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2jk\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890241 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890304 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890353 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890359 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890877 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890397 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.890877 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.890745 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.891128 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.891081 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.891185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.891167 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.892932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.892910 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.893804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.893788 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.906513 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.906417 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2jk\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.906513 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.906463 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token\") pod \"e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:54.936235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:54.935895 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:55.038359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.035013 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" event={"ID":"b0f71d83-d648-4a63-9289-60904622b040","Type":"ContainerStarted","Data":"a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a"} Nov 03 16:20:55.038359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.035136 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" event={"ID":"b0f71d83-d648-4a63-9289-60904622b040","Type":"ContainerStarted","Data":"4b15ce8fc163ae70cc0dd94479727bfaed194ad420ab07afbf36a5882e3bdf60"} Nov 03 16:20:55.144080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.143859 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podStartSLOduration=2.143838346 podStartE2EDuration="2.143838346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:55.069376432 +0000 UTC m=+1058.580744290" watchObservedRunningTime="2026-11-03 16:20:55.143838346 +0000 UTC m=+1058.655206216" Nov 03 16:20:55.147737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.147708 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff"] Nov 03 16:20:55.148341 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:55.148314 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8316fc_748b_4d59_9075_7bb4be758b7c.slice/crio-f80f158cc79592e6e905c21cf8f8110fb8cbaa457dd20bdd91c201a54106c607 WatchSource:0}: Error finding container f80f158cc79592e6e905c21cf8f8110fb8cbaa457dd20bdd91c201a54106c607: Status 404 returned error can't find the container with id f80f158cc79592e6e905c21cf8f8110fb8cbaa457dd20bdd91c201a54106c607 Nov 03 16:20:55.151115 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.151082 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:55.151197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.151147 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:55.151197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.151179 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:55.268923 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.268897 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p"] Nov 03 16:20:55.269665 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:20:55.269638 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6115cfdb_b425_4cff_a4ee_82e5452f98f3.slice/crio-a53f8c6c4049f940757e9bc73c7a5b2091de7dbed2b71098c558f5fd900ed0d5 WatchSource:0}: Error finding container a53f8c6c4049f940757e9bc73c7a5b2091de7dbed2b71098c558f5fd900ed0d5: Status 404 returned error can't find the container with id a53f8c6c4049f940757e9bc73c7a5b2091de7dbed2b71098c558f5fd900ed0d5 Nov 03 16:20:55.272636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.272603 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:55.272716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.272669 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:55.272716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.272696 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:55.344059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.334574 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:20:55.344059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.334718 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:20:55.344059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:55.334774 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:20:56.081278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.081237 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" event={"ID":"6115cfdb-b425-4cff-a4ee-82e5452f98f3","Type":"ContainerStarted","Data":"30aad3b85fdcb959db4130db50a5dc869210c6dc097603738a7197e470d7252a"} Nov 03 16:20:56.081790 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.081297 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" event={"ID":"6115cfdb-b425-4cff-a4ee-82e5452f98f3","Type":"ContainerStarted","Data":"a53f8c6c4049f940757e9bc73c7a5b2091de7dbed2b71098c558f5fd900ed0d5"} Nov 03 16:20:56.084906 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.084877 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" event={"ID":"fe8316fc-748b-4d59-9075-7bb4be758b7c","Type":"ContainerStarted","Data":"a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3"} Nov 03 16:20:56.085053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.084922 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" event={"ID":"fe8316fc-748b-4d59-9075-7bb4be758b7c","Type":"ContainerStarted","Data":"f80f158cc79592e6e905c21cf8f8110fb8cbaa457dd20bdd91c201a54106c607"} Nov 03 16:20:56.136008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.131417 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podStartSLOduration=2.131396085 podStartE2EDuration="2.131396085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.112488105 +0000 UTC m=+1059.623856000" watchObservedRunningTime="2026-11-03 16:20:56.131396085 +0000 UTC m=+1059.642763954" Nov 03 16:20:56.187929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.187863 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podStartSLOduration=2.187843221 podStartE2EDuration="2.187843221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:56.182219082 +0000 UTC m=+1059.693586953" watchObservedRunningTime="2026-11-03 16:20:56.187843221 +0000 UTC m=+1059.699211090" Nov 03 16:20:56.336067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.329633 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:20:56.336067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.329712 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:20:56.858012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.854341 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:20:56.858200 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.858096 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:20:56.858200 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.858152 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:20:56.947009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.942319 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:20:56.959011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.957116 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:20:56.959011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:56.957182 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:20:57.333009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:57.328399 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:20:57.333009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:57.328463 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:20:57.811700 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:20:57.811665 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:20:57.858011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:57.857167 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:20:57.858011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:57.857231 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:20:57.942010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:57.940467 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:20:57.942010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:57.940525 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:20:58.336056 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:58.336021 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:20:58.336640 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:58.336609 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:20:58.904007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:58.901651 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:20:58.904007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:58.901712 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:20:59.010217 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.010184 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:20:59.010380 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.010240 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:20:59.438006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.421852 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:20:59.438006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.423282 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:20:59.903601 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.903560 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:20:59.903800 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.903652 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:20:59.946645 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.946611 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:20:59.946954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:20:59.946908 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:21:00.041051 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.032116 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:00.339009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.332796 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:21:00.339009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.332854 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:21:00.938106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.931731 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:21:00.976691 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.976287 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:21:00.977164 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.939453 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:21:00.977164 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:00.977112 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:21:01.337408 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:01.337371 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:21:01.337693 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:01.337663 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:21:01.861025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:01.854512 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:21:01.861353 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:01.861313 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:21:01.975035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:01.970360 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:21:01.975035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:01.970422 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:21:02.328732 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:02.328390 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:21:02.328732 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:02.328449 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 16:21:02.858284 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:02.858249 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:21:02.858558 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:02.858525 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:21:02.951071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:02.950621 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:21:02.951071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:02.950683 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 16:21:03.373395 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:03.373364 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:21:03.376365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:03.376346 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:21:03.377663 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:03.377623 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:21:04.026096 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:04.026039 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:21:04.028465 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:04.028435 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:21:04.030612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:04.030595 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:21:04.173813 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:04.173776 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:21:04.174721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:04.174694 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:21:04.175664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:04.175645 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:21:10.256887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.255083 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.307457 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.307114 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.486011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.479695 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.508680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.507650 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:10.523350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.509965 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.523350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.510082 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.823202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.823095 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.850646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.850602 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:10.894098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:10.864516 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:11.262140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.262084 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:11.286080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.286050 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:21:11.674209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.674094 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 16:21:11.697453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.697409 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:11.698376 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.698353 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:21:11.698594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.698578 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:21:11.833008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.804921 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.834709 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.970343 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.970400 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.854464 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.865948 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.924087 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.969329 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.982093 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:21:12.031068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:11.982158 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:21:12.368063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.308887 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:12.727067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.725418 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:12.730076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.730040 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.973106 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.973147 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.973300 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.973532 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.973107 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.974236 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.975056 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:13.015422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:12.998376 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 16:21:13.418381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.418278 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:13.749292 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.746102 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:13.749292 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.748575 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 16:21:13.981133 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.981083 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:13.981743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.981714 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 16:21:13.982185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.982161 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:13.983785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.983439 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:13.984218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:13.983496 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:14.008294 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.007399 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 16:21:14.008294 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.007742 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 16:21:14.518060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.447111 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:14.518060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.447530 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:14.765011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.756335 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:14.765011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.760406 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:14.996047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.992612 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:14.996047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:14.992934 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:15.074021 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.058230 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:15.102026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.101218 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:15.180543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.161481 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:15.180543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.165172 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:15.180543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.176452 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm"] Nov 03 16:21:15.479382 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.459354 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:15.645009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.644284 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf"] Nov 03 16:21:15.808069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.802289 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 16:21:15.808069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:15.802242 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:16.049010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.040951 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:16.049010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.048649 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:16.100317 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.100148 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:16.101705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.101626 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:16.167151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.167110 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:16.182013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.181932 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 16:21:16.371059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.370567 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Nov 03 16:21:16.505974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.466381 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:21:16.506293 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.506273 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:21:16.506726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.506696 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:16.575482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.575444 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:21:16.575815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.575795 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:21:16.576166 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.576143 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:21:16.576929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.576794 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:21:16.576929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.576839 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:21:16.576929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.576869 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:21:16.578379 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:16.578357 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:21:23.997784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:23.997739 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js"] Nov 03 16:21:24.005609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.005581 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.018460 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.017570 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js"] Nov 03 16:21:24.141444 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141404 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141482 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927h6\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141553 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141587 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141742 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141706 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141743 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141766 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.141933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.141917 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.142023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.142006 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.243475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.243439 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.243638 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.243567 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.243695 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.243634 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.243758 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.243697 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.243926 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.243897 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927h6\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.243954 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244031 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244074 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244272 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244111 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244272 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244147 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244379 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244347 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244441 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244407 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244540 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244471 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.244680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.244660 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.246231 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.246206 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.246684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.246664 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.257265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.257209 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.257347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.257307 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927h6\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6\") pod \"e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.347319 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.347287 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:24.655573 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.655543 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js"] Nov 03 16:21:24.655721 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:24.655666 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd380c991_e43f_48a7_aa95_d4bd93207880.slice/crio-12560a731964ce0c2c85c993a9be6c936c470b7370ba33bcf048ab1baa4a23b5 WatchSource:0}: Error finding container 12560a731964ce0c2c85c993a9be6c936c470b7370ba33bcf048ab1baa4a23b5: Status 404 returned error can't find the container with id 12560a731964ce0c2c85c993a9be6c936c470b7370ba33bcf048ab1baa4a23b5 Nov 03 16:21:24.658504 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.658471 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:24.658620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.658542 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:24.658620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.658572 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:24.900512 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.900476 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj"] Nov 03 16:21:24.905188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.905124 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.923516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.923489 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj"] Nov 03 16:21:24.942904 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.942874 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp"] Nov 03 16:21:24.948783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.948754 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.967056 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.967028 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp"] Nov 03 16:21:24.995664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.995631 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.995899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.995876 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.996063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996045 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.996258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996238 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.996427 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996409 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.996571 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996554 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.996683 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996668 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.996799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996785 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.996906 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.996891 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.997064 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.997047 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhxj\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.997196 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.997180 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwx4\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.997335 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.997320 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.997551 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.997532 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.997726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.997708 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.997922 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.997890 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.998421 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.998400 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:24.998586 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.998569 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:24.998724 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:24.998686 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100209 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100243 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100248 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100273 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100314 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100369 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100398 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100445 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100468 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100499 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100594 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100668 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100696 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100732 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100753 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100776 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100776 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100816 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.100862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100849 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.101366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100911 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhxj\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.101366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100919 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.101366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.100949 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwx4\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.101366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.101046 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.101366 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.101220 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.101804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.101446 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.101804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.101558 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.101804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.101664 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.102050 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.102024 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.102215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.102170 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.103202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.103107 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.103772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.103750 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.104323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.104306 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.104660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.104638 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.115544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.115514 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwx4\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.115772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.115753 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.117613 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.117590 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.119250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.119213 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhxj\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj\") pod \"e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.228559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.228276 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:25.271138 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.271113 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:25.365114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.364600 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" event={"ID":"d380c991-e43f-48a7-aa95-d4bd93207880","Type":"ContainerStarted","Data":"377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903"} Nov 03 16:21:25.365114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.364667 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" event={"ID":"d380c991-e43f-48a7-aa95-d4bd93207880","Type":"ContainerStarted","Data":"12560a731964ce0c2c85c993a9be6c936c470b7370ba33bcf048ab1baa4a23b5"} Nov 03 16:21:25.391238 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.390342 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podStartSLOduration=2.3903238 podStartE2EDuration="2.3903238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:25.388940122 +0000 UTC m=+1088.900307987" watchObservedRunningTime="2026-11-03 16:21:25.3903238 +0000 UTC m=+1088.901691668" Nov 03 16:21:25.655646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.655597 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj"] Nov 03 16:21:25.681094 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.677277 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:25.681094 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.677370 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:25.681094 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.677511 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:25.682117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.681469 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp"] Nov 03 16:21:25.683405 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.682803 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:25.683405 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.682875 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:25.683405 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:25.682931 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:26.350008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.348328 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:26.355013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.354941 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:26.355145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.355025 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:26.405259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.405208 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" event={"ID":"c2451b7c-c27c-40b7-8e1f-ad322468b1d9","Type":"ContainerStarted","Data":"1b79ec7e98fc38638dbc8b291b3dd14fd6fd6c195134aaa812e073ea57019365"} Nov 03 16:21:26.405459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.405275 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" event={"ID":"c2451b7c-c27c-40b7-8e1f-ad322468b1d9","Type":"ContainerStarted","Data":"fceab93a0807ce35be07fc32d9eb32a57b071a043beb71b3bfa4573a829b59b0"} Nov 03 16:21:26.408175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.408139 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" event={"ID":"ab5e6618-836c-4f45-8ef3-7e9aaa552773","Type":"ContainerStarted","Data":"44b82474c2a9919f0ae441a09ec87f49277c742b983bccea582dbad4a5590a2b"} Nov 03 16:21:26.408298 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.408200 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" event={"ID":"ab5e6618-836c-4f45-8ef3-7e9aaa552773","Type":"ContainerStarted","Data":"b1b4581490c93fdab54283bfe3f5d020c0fd3bfc36c1d42f7be13816ec8cf121"} Nov 03 16:21:26.466009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.465250 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podStartSLOduration=2.465218782 podStartE2EDuration="2.465218782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:26.442734658 +0000 UTC m=+1089.954102537" watchObservedRunningTime="2026-11-03 16:21:26.465218782 +0000 UTC m=+1089.976586653" Nov 03 16:21:26.509009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:26.495657 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podStartSLOduration=2.4956355869999998 podStartE2EDuration="2.495635587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:26.477573309 +0000 UTC m=+1089.988941188" watchObservedRunningTime="2026-11-03 16:21:26.495635587 +0000 UTC m=+1090.007003457" Nov 03 16:21:27.234789 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.234745 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:27.235195 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.235159 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:27.235403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.235370 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:27.280008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.275047 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:27.287878 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.287297 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:27.287878 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.287354 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:27.447289 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.447252 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:27.474215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:27.474173 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:27.897269 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:21:27.897226 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:21:28.237009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:28.236542 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:28.237009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:28.236610 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:28.273009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:28.272374 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:28.273009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:28.272436 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:28.353183 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:28.353148 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:28.353436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:28.353412 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:29.231828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:29.231362 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:29.231828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:29.231428 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:29.274842 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:29.274408 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:29.274842 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:29.274466 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:29.361149 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:29.361113 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:29.361561 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:29.361529 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:30.259009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:30.245255 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:30.259009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:30.245314 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:30.287381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:30.287344 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:30.287621 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:30.287597 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:30.370011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:30.362494 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:30.370011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:30.362568 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:31.234030 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.233185 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:31.234030 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.233245 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:31.285114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.284591 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:31.285114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.284646 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:31.338630 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.338562 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" containerID="cri-o://a7fd66e1083e080ef258d55f7c834b61b2868216ebf7476deaeb71aec20c92f6" gracePeriod=44 Nov 03 16:21:31.381599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.381535 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:31.381884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.381855 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:31.509584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.505332 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" containerID="cri-o://bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68" gracePeriod=45 Nov 03 16:21:31.557033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.549261 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused" Nov 03 16:21:31.861821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.861388 6319 generic.go:356] "Generic (PLEG): container finished" podID="65133542-1e25-4611-906a-7aded0ed208b" containerID="a7fd66e1083e080ef258d55f7c834b61b2868216ebf7476deaeb71aec20c92f6" exitCode=0 Nov 03 16:21:31.861821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:31.861562 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" event={"ID":"65133542-1e25-4611-906a-7aded0ed208b","Type":"ContainerDied","Data":"a7fd66e1083e080ef258d55f7c834b61b2868216ebf7476deaeb71aec20c92f6"} Nov 03 16:21:32.203903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.202922 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Nov 03 16:21:32.208602 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.208559 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2"] Nov 03 16:21:32.239135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.233331 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:32.239135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.233386 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:32.297306 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.256338 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9"] Nov 03 16:21:32.297306 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.277670 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:32.297306 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.277722 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:32.311568 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.311029 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp"] Nov 03 16:21:32.424309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.409968 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:21:32.424309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.410056 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 16:21:32.595550 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.595511 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.651136 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir\") pod \"65133542-1e25-4611-906a-7aded0ed208b\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.651303 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache\") pod \"65133542-1e25-4611-906a-7aded0ed208b\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.651444 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs\" (UniqueName: \"kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs\") pod \"65133542-1e25-4611-906a-7aded0ed208b\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.651507 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7\" (UniqueName: \"kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7\") pod \"65133542-1e25-4611-906a-7aded0ed208b\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.651597 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm\") pod \"65133542-1e25-4611-906a-7aded0ed208b\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.651626 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home\") pod \"65133542-1e25-4611-906a-7aded0ed208b\" (UID: \"65133542-1e25-4611-906a-7aded0ed208b\") " Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.654041 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache" pod "65133542-1e25-4611-906a-7aded0ed208b" (UID: "65133542-1e25-4611-906a-7aded0ed208b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.654452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.654264 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home" pod "65133542-1e25-4611-906a-7aded0ed208b" (UID: "65133542-1e25-4611-906a-7aded0ed208b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.655029 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.654579 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir" pod "65133542-1e25-4611-906a-7aded0ed208b" (UID: "65133542-1e25-4611-906a-7aded0ed208b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.655634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.655572 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.655634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.655599 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.655634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.655612 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.681284 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.680224 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7" pod "65133542-1e25-4611-906a-7aded0ed208b" (UID: "65133542-1e25-4611-906a-7aded0ed208b"). InnerVolumeSpecName "kube-api-access-4lqz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:32.681963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.681917 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm" pod "65133542-1e25-4611-906a-7aded0ed208b" (UID: "65133542-1e25-4611-906a-7aded0ed208b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.708500 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.708469 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs" pod "65133542-1e25-4611-906a-7aded0ed208b" (UID: "65133542-1e25-4611-906a-7aded0ed208b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:32.710382 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.710359 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.762487 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs\") pod \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.762540 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache\") pod \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.762604 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir\") pod \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.762723 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm\" (UniqueName: \"kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm\") pod \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.762841 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home\") pod \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.763014 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm\") pod \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\" (UID: \"cdd53ee9-64ea-4c4b-b209-e4e9be14167d\") " Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.763777 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65133542-1e25-4611-906a-7aded0ed208b-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.763800 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65133542-1e25-4611-906a-7aded0ed208b-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.768005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.763817 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lqz7\" (UniqueName: \"kubernetes.io/projected/65133542-1e25-4611-906a-7aded0ed208b-kube-api-access-4lqz7\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.859173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.832318 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache" pod "cdd53ee9-64ea-4c4b-b209-e4e9be14167d" (UID: "cdd53ee9-64ea-4c4b-b209-e4e9be14167d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.859173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.850103 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs" pod "cdd53ee9-64ea-4c4b-b209-e4e9be14167d" (UID: "cdd53ee9-64ea-4c4b-b209-e4e9be14167d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:32.872038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.859850 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home" pod "cdd53ee9-64ea-4c4b-b209-e4e9be14167d" (UID: "cdd53ee9-64ea-4c4b-b209-e4e9be14167d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.872038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.870468 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.872038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.870504 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.872038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.870521 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.900642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.900577 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir" pod "cdd53ee9-64ea-4c4b-b209-e4e9be14167d" (UID: "cdd53ee9-64ea-4c4b-b209-e4e9be14167d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:32.991752 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.990359 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:32.998977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.995840 6319 generic.go:356] "Generic (PLEG): container finished" podID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerID="bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68" exitCode=0 Nov 03 16:21:32.998977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.995935 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" event={"ID":"cdd53ee9-64ea-4c4b-b209-e4e9be14167d","Type":"ContainerDied","Data":"bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68"} Nov 03 16:21:32.998977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.995968 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" event={"ID":"cdd53ee9-64ea-4c4b-b209-e4e9be14167d","Type":"ContainerDied","Data":"e8fd23302919c375a9fc0bb0423180cb1393d0e5bb258ab5e2ba0f4372daa0eb"} Nov 03 16:21:32.998977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.996019 6319 scope.go:122] "RemoveContainer" containerID="bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68" Nov 03 16:21:32.998977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:32.996184 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf" Nov 03 16:21:33.040223 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.025885 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" event={"ID":"65133542-1e25-4611-906a-7aded0ed208b","Type":"ContainerDied","Data":"e8738022a852f2fe1b31f55f4ff298f27b6eb625c686a4c4a1f34669dd826ba7"} Nov 03 16:21:33.041705 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.041676 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" Nov 03 16:21:33.165044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.163203 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm" pod "cdd53ee9-64ea-4c4b-b209-e4e9be14167d" (UID: "cdd53ee9-64ea-4c4b-b209-e4e9be14167d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:33.165044 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.163958 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm" pod "cdd53ee9-64ea-4c4b-b209-e4e9be14167d" (UID: "cdd53ee9-64ea-4c4b-b209-e4e9be14167d"). InnerVolumeSpecName "kube-api-access-rkhxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:33.167436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.167410 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhxm\" (UniqueName: \"kubernetes.io/projected/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-kube-api-access-rkhxm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:33.167646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.167629 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd53ee9-64ea-4c4b-b209-e4e9be14167d-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:33.170526 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.170505 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm"] Nov 03 16:21:33.207291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.207255 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm"] Nov 03 16:21:33.220345 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.220321 6319 scope.go:122] "RemoveContainer" containerID="bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68" Nov 03 16:21:33.221010 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:21:33.220963 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68\": container with ID starting with bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68 not found: ID does not exist" containerID="bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68" Nov 03 16:21:33.221258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.221229 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68"} err="failed to get container status \"bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68\": rpc error: code = NotFound desc = could not find container \"bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68\": container with ID starting with bdbd4511db656a8577ca733766384d49543b7324799fc62e98962c0637c83d68 not found: ID does not exist" Nov 03 16:21:33.221402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.221390 6319 scope.go:122] "RemoveContainer" containerID="a7fd66e1083e080ef258d55f7c834b61b2868216ebf7476deaeb71aec20c92f6" Nov 03 16:21:33.232230 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.232203 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:21:33.232349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.232267 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 16:21:33.272585 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.272554 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:21:33.272736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.272613 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 16:21:33.336691 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.336659 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf"] Nov 03 16:21:33.343565 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.343533 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d569fb/distinct-w0-d569fb-kserve-fd8ffb455-nplzf"] Nov 03 16:21:33.497494 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.497007 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d569fb/distinct-2-w0-d569fb-kserve-6b6b645fb8-rwmsm" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:33.599898 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.599864 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:33.601343 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.601276 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:33.602203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:33.602171 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:21:34.524320 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.524273 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:34.525310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.525288 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:34.527221 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.527197 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:34.631045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.631004 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:34.631848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.631824 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:34.633092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.633062 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:34.662037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.658255 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65133542-1e25-4611-906a-7aded0ed208b" path="/var/lib/kubelet/pods/65133542-1e25-4611-906a-7aded0ed208b/volumes" Nov 03 16:21:34.662037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.658997 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" path="/var/lib/kubelet/pods/cdd53ee9-64ea-4c4b-b209-e4e9be14167d/volumes" Nov 03 16:21:34.972063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:34.972033 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5"] Nov 03 16:21:36.745418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:36.745387 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd"] Nov 03 16:21:36.745917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:36.745643 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" podUID="66617af0-8b2f-4885-9c99-9b89707d3552" containerName="maas-api" containerID="cri-o://78217ca0aa18ec75b1bc0336feee4308ce3c091f45570425c8a0bd1afd5c2a67" gracePeriod=30 Nov 03 16:21:37.216664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.216500 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" containerID="cri-o://00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296" gracePeriod=25 Nov 03 16:21:37.311721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.311607 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" containerID="cri-o://f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245" gracePeriod=25 Nov 03 16:21:37.342597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.342534 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:21:37.358963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.352741 6319 generic.go:356] "Generic (PLEG): container finished" podID="66617af0-8b2f-4885-9c99-9b89707d3552" containerID="78217ca0aa18ec75b1bc0336feee4308ce3c091f45570425c8a0bd1afd5c2a67" exitCode=0 Nov 03 16:21:37.358963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.352811 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" event={"ID":"66617af0-8b2f-4885-9c99-9b89707d3552","Type":"ContainerDied","Data":"78217ca0aa18ec75b1bc0336feee4308ce3c091f45570425c8a0bd1afd5c2a67"} Nov 03 16:21:37.358963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.352853 6319 scope.go:122] "RemoveContainer" containerID="78217ca0aa18ec75b1bc0336feee4308ce3c091f45570425c8a0bd1afd5c2a67" Nov 03 16:21:37.358963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.353050 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd" Nov 03 16:21:37.438193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.438164 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls\") pod \"66617af0-8b2f-4885-9c99-9b89707d3552\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " Nov 03 16:21:37.438358 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.438244 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs\") pod \"66617af0-8b2f-4885-9c99-9b89707d3552\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " Nov 03 16:21:37.438513 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.438493 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca\") pod \"66617af0-8b2f-4885-9c99-9b89707d3552\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " Nov 03 16:21:37.438577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.438545 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx\" (UniqueName: \"kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx\") pod \"66617af0-8b2f-4885-9c99-9b89707d3552\" (UID: \"66617af0-8b2f-4885-9c99-9b89707d3552\") " Nov 03 16:21:37.438898 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.438872 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca" pod "66617af0-8b2f-4885-9c99-9b89707d3552" (UID: "66617af0-8b2f-4885-9c99-9b89707d3552"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:37.439222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.439197 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66617af0-8b2f-4885-9c99-9b89707d3552-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:37.441246 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.441222 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs" pod "66617af0-8b2f-4885-9c99-9b89707d3552" (UID: "66617af0-8b2f-4885-9c99-9b89707d3552"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:37.441339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.441298 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls" pod "66617af0-8b2f-4885-9c99-9b89707d3552" (UID: "66617af0-8b2f-4885-9c99-9b89707d3552"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:37.441593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.441574 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx" pod "66617af0-8b2f-4885-9c99-9b89707d3552" (UID: "66617af0-8b2f-4885-9c99-9b89707d3552"). InnerVolumeSpecName "kube-api-access-mk2sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:37.579435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.579351 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2sx\" (UniqueName: \"kubernetes.io/projected/66617af0-8b2f-4885-9c99-9b89707d3552-kube-api-access-mk2sx\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:37.579435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.579386 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:37.579435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.579402 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66617af0-8b2f-4885-9c99-9b89707d3552-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:37.689061 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.688939 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd"] Nov 03 16:21:37.700461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.700359 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d569fb-7958bb87c9-k7qjd"] Nov 03 16:21:37.988014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.982701 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:21:37.988014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:37.985858 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:21:38.680009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:38.672881 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66617af0-8b2f-4885-9c99-9b89707d3552" path="/var/lib/kubelet/pods/66617af0-8b2f-4885-9c99-9b89707d3552/volumes" Nov 03 16:21:39.133777 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.131412 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g"] Nov 03 16:21:39.149214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.149184 6319 cpu_manager.go:414] "containerMap: removing container" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" Nov 03 16:21:39.149426 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.149408 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" Nov 03 16:21:39.149554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.149540 6319 cpu_manager.go:414] "containerMap: removing container" podUID="66617af0-8b2f-4885-9c99-9b89707d3552" containerName="maas-api" Nov 03 16:21:39.149652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.149640 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66617af0-8b2f-4885-9c99-9b89707d3552" containerName="maas-api" Nov 03 16:21:39.149737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.149727 6319 cpu_manager.go:414] "containerMap: removing container" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" Nov 03 16:21:39.149809 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.149799 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" Nov 03 16:21:39.150117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.150102 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="66617af0-8b2f-4885-9c99-9b89707d3552" containerName="maas-api" Nov 03 16:21:39.150238 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.150227 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="65133542-1e25-4611-906a-7aded0ed208b" containerName="main" Nov 03 16:21:39.150354 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.150343 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd53ee9-64ea-4c4b-b209-e4e9be14167d" containerName="main" Nov 03 16:21:39.160775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.160755 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p"] Nov 03 16:21:39.161035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.160877 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.170114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.169170 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.182906 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.182878 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g"] Nov 03 16:21:39.187697 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.187677 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p"] Nov 03 16:21:39.218534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.218506 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9"] Nov 03 16:21:39.223818 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.223797 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.244939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.244910 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9"] Nov 03 16:21:39.261477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261453 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.261600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261501 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.261659 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261623 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.261732 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261719 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.261776 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261741 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdpw\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.261776 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261768 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.261853 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261830 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.261914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261901 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.262016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261930 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.262016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.261953 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.262093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262017 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.262125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262080 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.262169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262123 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.262169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262147 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.262235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262207 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.262235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262231 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.262294 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262272 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.262337 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.262302 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdlk9\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.272884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.272864 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7"] Nov 03 16:21:39.277214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.277198 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.339198 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.339170 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7"] Nov 03 16:21:39.363946 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.363911 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.364128 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.363972 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.364128 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364090 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.364276 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364131 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.364276 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364207 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.364276 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364245 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.364425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364306 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.364425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364377 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.364425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364396 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.364597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364430 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.364597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364505 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.364597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364533 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.364741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364612 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.364741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364615 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.364741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364682 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.364741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364738 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.364900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364776 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.364900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364801 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.364900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364813 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.365045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364914 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlk9\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.364954 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.365173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365070 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.365173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365100 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365124 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.365344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365221 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365275 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.365344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365305 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.365344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365330 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdpw\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.365480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365355 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365384 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.365480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365411 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkcwb\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.365480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365440 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.365600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365478 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmqw\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.365600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365505 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.365600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365536 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.365600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365564 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365600 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365584 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365761 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365643 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365761 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365671 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.365761 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365702 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365761 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365748 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.365875 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.365856 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.366166 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.366145 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.366261 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.366244 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.366501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.366457 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.367856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.367208 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.367856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.367566 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.368739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.368366 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.369315 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.369290 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.369431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.369355 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.387876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.387784 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.388051 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.387921 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlk9\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.394247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.394190 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.394595 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.394574 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdpw\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw\") pod \"e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.467689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.467640 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.467880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.467703 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.467880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.467762 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.467880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.467831 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.467936 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.467963 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkcwb\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468035 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468074 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmqw\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468105 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468139 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468263 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468317 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468372 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468396 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468671 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468428 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468671 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468461 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.468671 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468498 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.468671 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.468535 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.469900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.469267 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.469900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.469605 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.469900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.469630 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.469900 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.469875 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.470619 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.470835 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.471209 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.471545 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.471622 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.471675 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.471795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.471754 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.474262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.474238 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.476511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.476486 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.476618 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.476486 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.490390 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.490365 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.495643 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.495618 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkcwb\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.497019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.496963 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.499053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.499033 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:39.502051 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.502032 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmqw\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw\") pod \"e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.506959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.506937 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:39.539539 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.539497 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:39.594319 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.593771 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:39.656918 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.656837 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m"] Nov 03 16:21:39.823867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.823832 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g"] Nov 03 16:21:39.832960 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:39.832010 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6251865_47a9_408e_a9eb_9546ec0e4175.slice/crio-41a614c33314e02309d67a9249b111d94dec5435f09460db27927773e5c041b8 WatchSource:0}: Error finding container 41a614c33314e02309d67a9249b111d94dec5435f09460db27927773e5c041b8: Status 404 returned error can't find the container with id 41a614c33314e02309d67a9249b111d94dec5435f09460db27927773e5c041b8 Nov 03 16:21:39.834909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.834861 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.835037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.834959 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.835139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.835117 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.835590 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:39.835558 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3595af_163a_4339_858f_4f1885ce04a2.slice/crio-cbe91d70ad3ea93f8023a5cd9a3a31f6746495bc7dd4ba1e95f3feec87e49649 WatchSource:0}: Error finding container cbe91d70ad3ea93f8023a5cd9a3a31f6746495bc7dd4ba1e95f3feec87e49649: Status 404 returned error can't find the container with id cbe91d70ad3ea93f8023a5cd9a3a31f6746495bc7dd4ba1e95f3feec87e49649 Nov 03 16:21:39.837829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.837805 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p"] Nov 03 16:21:39.838368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.838319 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.838426 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.838384 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.838486 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.838432 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.886492 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.886433 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9"] Nov 03 16:21:39.893268 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:39.893236 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564b07ef_7d46_4fe2_b764_3edd43fba364.slice/crio-cfcba4d6a4ada311a6c8de2619b978f54ba66c5612ab8074b3774c4c58d3c1d0 WatchSource:0}: Error finding container cfcba4d6a4ada311a6c8de2619b978f54ba66c5612ab8074b3774c4c58d3c1d0: Status 404 returned error can't find the container with id cfcba4d6a4ada311a6c8de2619b978f54ba66c5612ab8074b3774c4c58d3c1d0 Nov 03 16:21:39.897611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.897571 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.897706 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.897677 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.897761 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.897730 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.936654 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.936622 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7"] Nov 03 16:21:39.943634 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:39.943601 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca99b6fc_2758_4f70_bd99_3728b13606b6.slice/crio-259a37a1655f72f0caba7aa04be962e29a3e3b9fa9b8f2546bdafccc5b6e5d2c WatchSource:0}: Error finding container 259a37a1655f72f0caba7aa04be962e29a3e3b9fa9b8f2546bdafccc5b6e5d2c: Status 404 returned error can't find the container with id 259a37a1655f72f0caba7aa04be962e29a3e3b9fa9b8f2546bdafccc5b6e5d2c Nov 03 16:21:39.952102 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.951690 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.952102 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.951771 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:39.952102 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:39.951800 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:21:40.563010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.559830 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" event={"ID":"564b07ef-7d46-4fe2-b764-3edd43fba364","Type":"ContainerStarted","Data":"c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860"} Nov 03 16:21:40.563010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.559890 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" event={"ID":"564b07ef-7d46-4fe2-b764-3edd43fba364","Type":"ContainerStarted","Data":"cfcba4d6a4ada311a6c8de2619b978f54ba66c5612ab8074b3774c4c58d3c1d0"} Nov 03 16:21:40.563010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.562328 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" event={"ID":"b6251865-47a9-408e-a9eb-9546ec0e4175","Type":"ContainerStarted","Data":"3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9"} Nov 03 16:21:40.563010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.562377 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" event={"ID":"b6251865-47a9-408e-a9eb-9546ec0e4175","Type":"ContainerStarted","Data":"41a614c33314e02309d67a9249b111d94dec5435f09460db27927773e5c041b8"} Nov 03 16:21:40.566139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.565567 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" event={"ID":"3d3595af-163a-4339-858f-4f1885ce04a2","Type":"ContainerStarted","Data":"ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f"} Nov 03 16:21:40.566139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.565613 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" event={"ID":"3d3595af-163a-4339-858f-4f1885ce04a2","Type":"ContainerStarted","Data":"cbe91d70ad3ea93f8023a5cd9a3a31f6746495bc7dd4ba1e95f3feec87e49649"} Nov 03 16:21:40.569182 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.569157 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" event={"ID":"ca99b6fc-2758-4f70-bd99-3728b13606b6","Type":"ContainerStarted","Data":"ce90efc4493f3b43b3a3f1524d66e512840eb5bb7324b98e409a704b56924477"} Nov 03 16:21:40.569469 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.569417 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" event={"ID":"ca99b6fc-2758-4f70-bd99-3728b13606b6","Type":"ContainerStarted","Data":"259a37a1655f72f0caba7aa04be962e29a3e3b9fa9b8f2546bdafccc5b6e5d2c"} Nov 03 16:21:40.601960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.601928 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:40.606008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.604628 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podStartSLOduration=1.604608249 podStartE2EDuration="1.604608249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.601673761 +0000 UTC m=+1104.113041630" watchObservedRunningTime="2026-11-03 16:21:40.604608249 +0000 UTC m=+1104.115976117" Nov 03 16:21:40.964012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.963210 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podStartSLOduration=1.963185049 podStartE2EDuration="1.963185049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.658440474 +0000 UTC m=+1104.169808343" watchObservedRunningTime="2026-11-03 16:21:40.963185049 +0000 UTC m=+1104.474552917" Nov 03 16:21:40.979014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:40.964339 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podStartSLOduration=1.96432108 podStartE2EDuration="1.96432108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.696454956 +0000 UTC m=+1104.207822824" watchObservedRunningTime="2026-11-03 16:21:40.96432108 +0000 UTC m=+1104.475688948" Nov 03 16:21:41.089014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.088824 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podStartSLOduration=2.088800454 podStartE2EDuration="2.088800454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:21:40.929572854 +0000 UTC m=+1104.440940725" watchObservedRunningTime="2026-11-03 16:21:41.088800454 +0000 UTC m=+1104.600168325" Nov 03 16:21:41.520010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.513644 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:41.520010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.513850 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:41.520010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.513904 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:41.626139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.590052 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:41.626139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.592144 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:41.626139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.617149 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:41.626139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.617208 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:41.662076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.638699 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:41.662076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.638754 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:41.743005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.670690 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:41.743005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.670762 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:41.790919 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.771347 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:41.996016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.992136 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:41.996016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.993581 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:41.996016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:41.993818 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:42.041063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.011369 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:42.395192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.395088 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:42.501496 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.501459 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:42.501793 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.501752 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:42.512004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.511954 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:42.512270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.512236 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:42.545026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.544973 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:42.545323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.545284 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:42.595876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.595396 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:42.595876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.595458 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:42.775011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.774668 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:42.887013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:42.884768 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:21:43.081062 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.060487 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:43.081062 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.062555 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:43.081062 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.062959 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:43.081062 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.063269 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:21:43.081062 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.063324 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:21:43.160025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.088559 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:43.160025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.088931 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:21:43.160025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.089057 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:43.160025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.089402 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:21:43.431106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.402409 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:43.478532 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.478503 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:21:43.586151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.565200 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:43.586151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.565263 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:43.586407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.586264 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:43.586407 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.586312 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:43.606017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.588226 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:43.627131 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.627085 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:43.899093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.785012 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:43.899093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.830894 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:43.899093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:43.830953 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:44.106011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.083592 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:44.106011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.084062 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:44.106608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.106556 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:44.107209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.107172 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:44.124126 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.108938 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:21:44.149010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.111070 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:44.407019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.406713 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:44.618010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.567959 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:44.618010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.568040 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:44.618010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.568437 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:44.618010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.568475 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:44.618010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.593152 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:44.618010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.593206 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:44.618903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.618874 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:44.619335 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.619308 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:44.796466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:44.795018 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:45.098746 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.098630 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:45.195927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.137679 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:45.198527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.196931 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:45.217339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.203031 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:45.667106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.667072 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:45.667429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.667380 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:45.734012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.722862 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:45.975005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.974934 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:45.988045 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.824947 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp"] Nov 03 16:21:45.989037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.988972 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" containerID="cri-o://1b79ec7e98fc38638dbc8b291b3dd14fd6fd6c195134aaa812e073ea57019365" gracePeriod=30 Nov 03 16:21:45.989409 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.832817 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:46.031015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.944098 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:46.031384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.031346 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:46.031611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.807138 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:46.032157 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:45.944184 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:46.033267 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.032344 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:46.033267 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.033061 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:46.192144 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.192100 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:46.219642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.196109 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:46.219642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.198367 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4"] Nov 03 16:21:46.219642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.198586 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" containerID="cri-o://6616e8c1e0f6148b01bc33f12fcb797bfc85249bb27fc6a5635e5820c1057d42" gracePeriod=30 Nov 03 16:21:46.219642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.199072 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:46.228063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.220883 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:46.299255 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.299208 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj"] Nov 03 16:21:46.308185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.308132 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" containerID="cri-o://44b82474c2a9919f0ae441a09ec87f49277c742b983bccea582dbad4a5590a2b" gracePeriod=30 Nov 03 16:21:46.364009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.356738 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt"] Nov 03 16:21:46.364512 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.364474 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" containerID="cri-o://e4f9e6c52cdd2dc9e125fba77c77f1ef23a245c94a1b8e997fc27b6a897a1449" gracePeriod=30 Nov 03 16:21:46.585107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.553409 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:46.585107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.553503 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:46.585107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.555492 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:46.585107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.555535 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:46.585107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.555785 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:46.585107 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.555816 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:46.618135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.618099 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:46.618403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:46.618371 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:47.021428 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.021385 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:47.136058 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.131072 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:47.136058 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.131452 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:47.205711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.205655 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:47.241480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.241424 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:47.299146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.248259 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:47.499555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.308884 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 16:21:47.501654 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.317681 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:47.502549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.502525 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:47.502751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.502725 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:47.560285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.560179 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:47.560560 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.560513 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:47.560782 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.560765 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:47.560929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.560906 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:47.657161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.657129 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:47.657417 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:47.657389 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:48.005915 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.005864 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:21:48.037194 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.037142 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Nov 03 16:21:48.037939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.037897 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:21:48.096283 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.096238 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.124922 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.124871 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:48.125188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.125146 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.170839 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.170799 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 03 16:21:48.188108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.172177 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:48.212971 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.212938 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:48.213554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.213523 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.216574 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.216536 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:48.216780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.216758 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.243433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.243375 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.292112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.290482 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:48.697097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.692240 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.698280 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.698247 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:48.698506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.698462 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:48.731032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.722255 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:48.731032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.722316 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.772123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.732013 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" containerID="cri-o://e29d73cbf9757619acc7e162052b77749fbb2019a2e9ecb3bfca1b028214430c" gracePeriod=44 Nov 03 16:21:48.772123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.745011 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:48.772123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.745078 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:48.772123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.749072 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.778013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.777961 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:48.790052 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.779798 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:48.839053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.785066 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:48.839053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.813297 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:48.876552 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.876500 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:48.876927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.876853 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:48.955539 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.955442 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:21:48.955820 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:48.955780 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:21:49.108014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.106079 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:49.205039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.199703 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.205039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.201163 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.264169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.264096 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.293512 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.293445 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.401357 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.401299 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": read tcp 10.134.0.2:51488->10.134.0.71:8000: read: connection reset by peer" Nov 03 16:21:49.448053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.448013 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:49.448337 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.448305 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.507011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.506097 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:49.507011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.506159 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:49.532069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.513724 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:49.532069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.513783 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:49.557012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.548061 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:49.557012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.548140 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:49.557012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.548293 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:21:49.557012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.548363 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:21:49.620012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.619765 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:49.620012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.619827 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:49.645135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.644356 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:21:49.645135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.644415 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:21:49.696135 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.696080 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.727148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.727107 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:49.727428 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.727400 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.728226 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.728192 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:49.753394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.753351 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:49.835601 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.835211 6319 patch_prober.go:29] interesting pod/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:21:49.835601 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.835274 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:21:49.913060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.871030 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:49.913060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.871099 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:49.913060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.891092 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:49.913060 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:49.891160 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.115278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.108087 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:50.115278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.108162 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.115278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.108507 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.298136 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.298086 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.312921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.299319 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.312921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.300959 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 03 16:21:50.314571 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.314514 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.485022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.477659 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" containerID="cri-o://409710093bc6da7f4c532f14624bc119aeca75793ee9b63ff6fc8dfac84a4592" gracePeriod=45 Nov 03 16:21:50.567011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.510082 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:50.567011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.536117 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.568036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.510527 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:50.568269 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.568237 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:50.583335 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.583309 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:50.583544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.583523 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:50.599563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.596183 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:50.638389 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.638352 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:50.638651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.638626 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:50.716013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.703128 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.784685 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.754942 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:50.937140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:50.937098 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:51.159076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.117130 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:51.303063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.302966 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:51.317459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.304173 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:51.356604 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.356564 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": read tcp 10.134.0.2:50670->10.134.0.64:8000: read: connection reset by peer" Nov 03 16:21:51.379005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.370488 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:51.472888 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.472841 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:51.517011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.505147 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:51.517011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.505204 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:51.535018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.534119 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:51.535018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.534182 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:51.565025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.563928 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:51.565025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.564004 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:51.668171 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.668135 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:51.689703 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.689660 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:51.729466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.729365 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:51.756122 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:51.756069 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:52.014748 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.014643 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:52.143025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.137398 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:52.353609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.320242 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:52.387043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.355902 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:52.482089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.480551 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:52.557543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.557506 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:52.557785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.557756 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:52.769066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.754566 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:52.769066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.754734 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:52.771690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.771660 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:52.771924 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.771895 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:52.827013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.804555 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:52.848006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.820940 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:52.848006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.821692 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:52.848006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:52.835350 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:53.019410 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.019309 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:53.141133 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.141084 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:53.168009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.159110 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4"] Nov 03 16:21:53.215010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.168478 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" podUID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" containerName="maas-api" containerID="cri-o://0f979ba2e4010ada07b3e85e833f2d740c58e3e95d56dd7ab428e59e1d55a38f" gracePeriod=30 Nov 03 16:21:53.400334 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.280681 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:21:53.439222 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.323138 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:53.513081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.497879 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:53.513597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.510386 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:53.532120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.532070 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:53.532379 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.511490 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:53.532492 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.532471 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:53.630578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.627528 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:53.630578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.627586 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:53.682011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.632076 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:53.682011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.632145 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:53.815972 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.815936 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:53.846382 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:53.846261 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:54.190236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.000509 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:54.190236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.143061 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:54.337664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.329345 6319 generic.go:356] "Generic (PLEG): container finished" podID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerID="e29d73cbf9757619acc7e162052b77749fbb2019a2e9ecb3bfca1b028214430c" exitCode=0 Nov 03 16:21:54.337664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.329615 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" event={"ID":"1605541c-2e9c-42fd-9cd0-1606e0606e83","Type":"ContainerDied","Data":"e29d73cbf9757619acc7e162052b77749fbb2019a2e9ecb3bfca1b028214430c"} Nov 03 16:21:54.459119 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.330305 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:54.499511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.499453 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:21:54.595098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.595049 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:54.623918 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.623887 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:21:54.690048 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.677598 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:21:54.695029 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.690319 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:21:54.708397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.697161 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:54.708397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.697240 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:54.708397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.702316 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:54.708397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.702387 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:54.708397 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.702892 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:54.751091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.749327 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:54.751091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.749485 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:54.751091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.749534 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:54.751091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.749613 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:54.751091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.749651 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:54.810855 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:54.810183 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" containerID="cri-o://23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b" gracePeriod=45 Nov 03 16:21:55.093019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.067465 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:55.191377 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.156639 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:55.191377 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.188007 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:55.447386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.431884 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:55.628928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.627412 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:55.628928 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.627485 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:55.671914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.667465 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:55.671914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.667529 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:55.933285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.926343 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:55.933285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:55.926508 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection reset by peer" Nov 03 16:21:56.406008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.398638 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:21:56.454510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.437925 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:56.479099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.471179 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:56.520189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.509188 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:56.535021 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.533417 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:56.628048 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.626085 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" podUID="c79ffcfe-b586-4ebf-a29e-f2f6c155784c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:56.662010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.661797 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 03 16:21:56.779070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.778276 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" podUID="bed9f35a-eb7a-4d7d-b328-b5fa380c8676" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:56.866517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.833829 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:56.866517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.833905 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:56.891740 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:56.891677 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg" podUID="634e1091-c4df-49da-9b69-037e924c624e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:57.003485 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.003441 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 03 16:21:57.032070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.025259 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:57.032070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.025348 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:57.058036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.053743 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:57.058036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.053816 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:57.339899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.324531 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:57.339899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.324589 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:57.434879 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.434830 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:57.461167 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.450179 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" podUID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.53:8443/health\": dial tcp 10.134.0.53:8443: connect: connection refused" Nov 03 16:21:57.497298 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.485283 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:21:57.525023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.517095 6319 patch_prober.go:29] interesting pod/console-565565c4b5-zqhq2 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:21:57.525023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.517168 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-565565c4b5-zqhq2" podUID="c8b2287e-d813-4fb2-bea3-5443013af667" containerName="console" probeResult="failure" output="Get \"https://10.134.0.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:57.525023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.518588 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-j7zvw" podUID="440b9503-cf4d-4749-94b7-eb4101c00c59" containerName="server" probeResult="failure" output="Get \"https://10.134.0.24:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:57.593324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.580085 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:57.593324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.580154 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:57.638134 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.638090 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:21:57.700640 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.700601 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:21:57.780141 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.780079 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" podUID="bed9f35a-eb7a-4d7d-b328-b5fa380c8676" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:57.781153 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.781114 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" podUID="bed9f35a-eb7a-4d7d-b328-b5fa380c8676" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:57.843076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.839568 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:57.843076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:57.839631 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:58.003147 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.003069 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Nov 03 16:21:58.180243 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:21:58.180209 6319 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.404s" Nov 03 16:21:58.183945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.183922 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:21:58.185615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.185595 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:21:58.192295 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.192243 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:58.192747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.192721 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 03 16:21:58.193204 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.193182 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.193360 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.193342 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:58.193505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.193491 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.193609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.193594 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:58.193757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.193743 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.193877 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.193861 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:58.194035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.194020 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.194158 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.194142 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:58.222283 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.221227 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 03 16:21:58.222283 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.221340 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:58.317465 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:58.292519 6319 logging.go:55] [core] [Channel #506 SubChannel #507]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:50052", ServerName: "10.134.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:50052: connect: connection refused" Nov 03 16:21:58.432269 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.430554 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" podUID="c79ffcfe-b586-4ebf-a29e-f2f6c155784c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:58.537355 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.520675 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.537355 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.520754 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:58.537355 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.520852 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.537355 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.520898 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:58.544777 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:58.544750 6319 logging.go:55] [core] [Channel #508 SubChannel #509]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: connect: connection refused" Nov 03 16:21:58.565005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.564196 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.565005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.564260 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:58.596175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.596078 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:58.596175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.596151 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:58.624800 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:21:58.620392 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:21:58.624800 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.620811 6319 generic.go:356] "Generic (PLEG): container finished" podID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerID="6616e8c1e0f6148b01bc33f12fcb797bfc85249bb27fc6a5635e5820c1057d42" exitCode=0 Nov 03 16:21:58.628677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.628645 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" event={"ID":"11a0bf2d-0d7c-457c-9be4-b6e08db98c91","Type":"ContainerDied","Data":"6616e8c1e0f6148b01bc33f12fcb797bfc85249bb27fc6a5635e5820c1057d42"} Nov 03 16:21:58.629052 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.629034 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4"] Nov 03 16:21:58.631159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.631131 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:21:58.632453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.632400 6319 scope.go:122] "RemoveContainer" containerID="179c5c6d9c3937f1455c520a9a9aae620c3abc604691b56e5bf1fc159587c34b" Nov 03 16:21:58.647947 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.647751 6319 generic.go:356] "Generic (PLEG): container finished" podID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerID="1b79ec7e98fc38638dbc8b291b3dd14fd6fd6c195134aaa812e073ea57019365" exitCode=0 Nov 03 16:21:58.648561 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.648275 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:21:58.648561 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.648334 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:21:58.648561 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.648365 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4"] Nov 03 16:21:58.650300 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.649834 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:58.655325 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.655211 6319 generic.go:356] "Generic (PLEG): container finished" podID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerID="44b82474c2a9919f0ae441a09ec87f49277c742b983bccea582dbad4a5590a2b" exitCode=0 Nov 03 16:21:58.659359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.659324 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:21:58.659670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.659656 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rqgdz\"" Nov 03 16:21:58.728616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.727219 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs\" (UniqueName: \"kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs\") pod \"1605541c-2e9c-42fd-9cd0-1606e0606e83\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " Nov 03 16:21:58.728616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.727287 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm\") pod \"1605541c-2e9c-42fd-9cd0-1606e0606e83\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " Nov 03 16:21:58.728616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.727363 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh\" (UniqueName: \"kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh\") pod \"1605541c-2e9c-42fd-9cd0-1606e0606e83\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " Nov 03 16:21:58.728616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.727503 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home\") pod \"1605541c-2e9c-42fd-9cd0-1606e0606e83\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " Nov 03 16:21:58.728616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.727543 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir\") pod \"1605541c-2e9c-42fd-9cd0-1606e0606e83\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " Nov 03 16:21:58.728616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.727589 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache\") pod \"1605541c-2e9c-42fd-9cd0-1606e0606e83\" (UID: \"1605541c-2e9c-42fd-9cd0-1606e0606e83\") " Nov 03 16:21:58.729479 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.729452 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache" pod "1605541c-2e9c-42fd-9cd0-1606e0606e83" (UID: "1605541c-2e9c-42fd-9cd0-1606e0606e83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:58.729852 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.729819 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home" pod "1605541c-2e9c-42fd-9cd0-1606e0606e83" (UID: "1605541c-2e9c-42fd-9cd0-1606e0606e83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:58.730236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.730199 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir" pod "1605541c-2e9c-42fd-9cd0-1606e0606e83" (UID: "1605541c-2e9c-42fd-9cd0-1606e0606e83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805073 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" event={"ID":"c2451b7c-c27c-40b7-8e1f-ad322468b1d9","Type":"ContainerDied","Data":"1b79ec7e98fc38638dbc8b291b3dd14fd6fd6c195134aaa812e073ea57019365"} Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805154 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" event={"ID":"ab5e6618-836c-4f45-8ef3-7e9aaa552773","Type":"ContainerDied","Data":"44b82474c2a9919f0ae441a09ec87f49277c742b983bccea582dbad4a5590a2b"} Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805264 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805289 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l4d5b" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805344 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805389 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805434 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805470 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ggwbh" Nov 03 16:21:58.807794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.805508 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:21:58.869072 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.839922 6319 generic.go:356] "Generic (PLEG): container finished" podID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" containerID="0f979ba2e4010ada07b3e85e833f2d740c58e3e95d56dd7ab428e59e1d55a38f" exitCode=0 Nov 03 16:21:58.869072 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.840037 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" event={"ID":"def1b0ba-39a7-47ed-9dcd-bc84b2d14410","Type":"ContainerDied","Data":"0f979ba2e4010ada07b3e85e833f2d740c58e3e95d56dd7ab428e59e1d55a38f"} Nov 03 16:21:58.869072 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.843404 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh" pod "1605541c-2e9c-42fd-9cd0-1606e0606e83" (UID: "1605541c-2e9c-42fd-9cd0-1606e0606e83"). InnerVolumeSpecName "kube-api-access-sw4kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:58.869072 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.843477 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm" pod "1605541c-2e9c-42fd-9cd0-1606e0606e83" (UID: "1605541c-2e9c-42fd-9cd0-1606e0606e83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:58.869072 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.843525 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs" pod "1605541c-2e9c-42fd-9cd0-1606e0606e83" (UID: "1605541c-2e9c-42fd-9cd0-1606e0606e83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:58.972038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.885164 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:21:58.972038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.899397 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:58.972038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.899769 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7fw\" (UniqueName: \"kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:58.972038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.899961 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:58.972038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:58.937751 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.008932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.008291 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 16:21:59.008932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.008563 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 03 16:21:59.009974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.009590 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.009974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.009652 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.009974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.009890 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605541c-2e9c-42fd-9cd0-1606e0606e83-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.013932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.013605 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:21:59.014505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.014094 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605541c-2e9c-42fd-9cd0-1606e0606e83-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.014505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.014119 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw4kh\" (UniqueName: \"kubernetes.io/projected/1605541c-2e9c-42fd-9cd0-1606e0606e83-kube-api-access-sw4kh\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.023005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.021167 6319 generic.go:356] "Generic (PLEG): container finished" podID="6d44abb0-b332-4714-9585-0be6d576d530" containerID="e4f9e6c52cdd2dc9e125fba77c77f1ef23a245c94a1b8e997fc27b6a897a1449" exitCode=0 Nov 03 16:21:59.023005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.021282 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" event={"ID":"6d44abb0-b332-4714-9585-0be6d576d530","Type":"ContainerDied","Data":"e4f9e6c52cdd2dc9e125fba77c77f1ef23a245c94a1b8e997fc27b6a897a1449"} Nov 03 16:21:59.034422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.034393 6319 generic.go:356] "Generic (PLEG): container finished" podID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerID="409710093bc6da7f4c532f14624bc119aeca75793ee9b63ff6fc8dfac84a4592" exitCode=0 Nov 03 16:21:59.037161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.034610 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" event={"ID":"86b42ef8-935c-4a0d-b01e-848f3bc493bb","Type":"ContainerDied","Data":"409710093bc6da7f4c532f14624bc119aeca75793ee9b63ff6fc8dfac84a4592"} Nov 03 16:21:59.068431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.068408 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.118809 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.118880 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.118962 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119018 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119056 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7\" (UniqueName: \"kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7\") pod \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119149 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\") pod \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119196 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119237 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119275 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca\") pod \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119411 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs\") pod \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\" (UID: \"def1b0ba-39a7-47ed-9dcd-bc84b2d14410\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119453 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119512 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119555 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj\") pod \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\" (UID: \"ab5e6618-836c-4f45-8ef3-7e9aaa552773\") " Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.119776 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.120032 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7fw\" (UniqueName: \"kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.120329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.120152 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.121324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.120444 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.121324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.120582 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.124602 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.124015 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.124602 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.124343 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.125798 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.125287 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.128363 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.128162 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:21:59.128860 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.128600 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca" pod "def1b0ba-39a7-47ed-9dcd-bc84b2d14410" (UID: "def1b0ba-39a7-47ed-9dcd-bc84b2d14410"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.130367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.130345 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.130747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.130726 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.131963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.131915 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "kube-api-access-wwhxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.133548 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.133524 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs" pod "def1b0ba-39a7-47ed-9dcd-bc84b2d14410" (UID: "def1b0ba-39a7-47ed-9dcd-bc84b2d14410"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:59.138458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.138417 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.140347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.140307 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7" pod "def1b0ba-39a7-47ed-9dcd-bc84b2d14410" (UID: "def1b0ba-39a7-47ed-9dcd-bc84b2d14410"). InnerVolumeSpecName "kube-api-access-cqhc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.141489 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.141453 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.149026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.148978 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy" pod "ab5e6618-836c-4f45-8ef3-7e9aaa552773" (UID: "ab5e6618-836c-4f45-8ef3-7e9aaa552773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.157727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.156745 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7fw\" (UniqueName: \"kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw\") pod \"maas-api-revoke-sub-789180a8-gnjl4\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.165918 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.163905 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls" pod "def1b0ba-39a7-47ed-9dcd-bc84b2d14410" (UID: "def1b0ba-39a7-47ed-9dcd-bc84b2d14410"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:59.195858 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.184605 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.231835 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.231900 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232008 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232060 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232180 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232251 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232343 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232463 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.232538 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket\") pod \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\" (UID: \"c2451b7c-c27c-40b7-8e1f-ad322468b1d9\") " Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233669 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233693 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233716 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233731 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhxj\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-kube-api-access-wwhxj\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233747 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233761 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233780 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233798 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqhc7\" (UniqueName: \"kubernetes.io/projected/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-kube-api-access-cqhc7\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233812 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233831 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233847 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e6618-836c-4f45-8ef3-7e9aaa552773-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.241511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.233863 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def1b0ba-39a7-47ed-9dcd-bc84b2d14410-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.242714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.242489 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:21:59.242772 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.242724 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.243796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.243759 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.244122 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.244092 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.244652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.244626 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.244816 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.244773 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.246259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.246216 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "kube-api-access-9nwx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.248100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.247354 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.257085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.257032 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token" pod "c2451b7c-c27c-40b7-8e1f-ad322468b1d9" (UID: "c2451b7c-c27c-40b7-8e1f-ad322468b1d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.273256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.273215 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:50052\" within 1s: context deadline exceeded" Nov 03 16:21:59.273608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.273574 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:21:59.338557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.338519 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339322 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339301 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339459 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339583 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339570 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339672 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339773 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339762 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339855 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwx4\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-kube-api-access-9nwx4\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.339954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.339943 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.340067 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.340053 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2451b7c-c27c-40b7-8e1f-ad322468b1d9-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.443316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.443151 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:21:59.448678 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.443647 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded" Nov 03 16:21:59.448678 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.448506 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:21:59.541539 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.541508 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:21:59.545241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.541741 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 16:21:59.545241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.543336 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:21:59.545241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.543369 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 16:21:59.545241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.545218 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 16:21:59.545544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.545256 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 16:21:59.596073 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.595695 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:21:59.607158 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.607119 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:21:59.608121 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.607379 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:21:59.630756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.630409 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:21:59.687058 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.672214 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:21:59.691315 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.691287 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:21:59.707240 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.707157 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:21:59.758447 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.758296 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:21:59.786047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786013 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.786290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786273 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.786451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786415 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.786572 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786560 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.786701 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786688 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.786795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786784 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.786956 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.786942 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.787093 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787079 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.787242 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787229 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.787359 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787347 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache\") pod \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " Nov 03 16:21:59.787480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787468 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.787578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787567 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.787695 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787682 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.787925 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.787913 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.788055 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788038 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk\" (UniqueName: \"kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk\") pod \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " Nov 03 16:21:59.788193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788181 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.788329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788314 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home\") pod \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " Nov 03 16:21:59.788428 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788416 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.788525 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788514 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm\") pod \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " Nov 03 16:21:59.788651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788636 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8\") pod \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\" (UID: \"11a0bf2d-0d7c-457c-9be4-b6e08db98c91\") " Nov 03 16:21:59.788834 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.788821 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir\") pod \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " Nov 03 16:21:59.792173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.792149 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.792486 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.792470 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home\") pod \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " Nov 03 16:21:59.792781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.792767 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm\") pod \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " Nov 03 16:21:59.792910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.792897 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs\") pod \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\" (UID: \"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5\") " Nov 03 16:21:59.793188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.793173 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket\") pod \"6d44abb0-b332-4714-9585-0be6d576d530\" (UID: \"6d44abb0-b332-4714-9585-0be6d576d530\") " Nov 03 16:21:59.793496 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.793481 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv\" (UniqueName: \"kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv\") pod \"988c231f-9d8e-4873-8e01-8599e5d904d6\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.790124 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.790390 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.791075 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.791443 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.791665 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home" pod "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" (UID: "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.792601 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache" pod "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" (UID: "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.793313 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.794723 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.810301 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.795120 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.799596 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.802339 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.809332 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.809937 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home" pod "86b42ef8-935c-4a0d-b01e-848f3bc493bb" (UID: "86b42ef8-935c-4a0d-b01e-848f3bc493bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.810150 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir" pod "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" (UID: "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.810894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.810219 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm" pod "86b42ef8-935c-4a0d-b01e-848f3bc493bb" (UID: "86b42ef8-935c-4a0d-b01e-848f3bc493bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.813120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.813081 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "kube-api-access-cvr9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.813415 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.813388 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "kube-api-access-xm8t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.822597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.822546 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.822862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.822823 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.830643 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.830584 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:21:59.831611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.831568 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv" pod "988c231f-9d8e-4873-8e01-8599e5d904d6" (UID: "988c231f-9d8e-4873-8e01-8599e5d904d6"). InnerVolumeSpecName "kube-api-access-n58tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.832442 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.832406 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs" pod "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" (UID: "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:59.832677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.832644 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk" pod "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" (UID: "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5"). InnerVolumeSpecName "kube-api-access-wcjvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.832949 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.832893 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo" pod "6d44abb0-b332-4714-9585-0be6d576d530" (UID: "6d44abb0-b332-4714-9585-0be6d576d530"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:21:59.832949 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.832906 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy" pod "11a0bf2d-0d7c-457c-9be4-b6e08db98c91" (UID: "11a0bf2d-0d7c-457c-9be4-b6e08db98c91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.833236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.833204 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm" pod "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" (UID: "4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895069 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs\") pod \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895184 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl\" (UniqueName: \"kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl\") pod \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895245 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir\") pod \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895309 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume\") pod \"38c01d5a-fc60-4aee-89be-50ecd3abb448\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895412 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume\") pod \"988c231f-9d8e-4873-8e01-8599e5d904d6\" (UID: \"988c231f-9d8e-4873-8e01-8599e5d904d6\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895466 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache\") pod \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\" (UID: \"86b42ef8-935c-4a0d-b01e-848f3bc493bb\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.895559 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp\" (UniqueName: \"kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp\") pod \"38c01d5a-fc60-4aee-89be-50ecd3abb448\" (UID: \"38c01d5a-fc60-4aee-89be-50ecd3abb448\") " Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896327 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896353 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896368 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896383 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm8t8\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-kube-api-access-xm8t8\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896396 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896409 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d44abb0-b332-4714-9585-0be6d576d530-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896423 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896436 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896448 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896460 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896474 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58tv\" (UniqueName: \"kubernetes.io/projected/988c231f-9d8e-4873-8e01-8599e5d904d6-kube-api-access-n58tv\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896486 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896498 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896510 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896521 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896534 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvr9b\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-kube-api-access-cvr9b\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896546 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896559 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d44abb0-b332-4714-9585-0be6d576d530-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896570 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896582 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896595 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896607 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896621 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896633 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a0bf2d-0d7c-457c-9be4-b6e08db98c91-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896645 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d44abb0-b332-4714-9585-0be6d576d530-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896660 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcjvk\" (UniqueName: \"kubernetes.io/projected/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5-kube-api-access-wcjvk\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.897945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.896673 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d44abb0-b332-4714-9585-0be6d576d530-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:21:59.928130 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.915808 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume" pod "38c01d5a-fc60-4aee-89be-50ecd3abb448" (UID: "38c01d5a-fc60-4aee-89be-50ecd3abb448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.928681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.928403 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache" pod "86b42ef8-935c-4a0d-b01e-848f3bc493bb" (UID: "86b42ef8-935c-4a0d-b01e-848f3bc493bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.928681 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.928644 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume" pod "988c231f-9d8e-4873-8e01-8599e5d904d6" (UID: "988c231f-9d8e-4873-8e01-8599e5d904d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:21:59.928961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.928925 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir" pod "86b42ef8-935c-4a0d-b01e-848f3bc493bb" (UID: "86b42ef8-935c-4a0d-b01e-848f3bc493bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:21:59.936780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.936755 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp" pod "38c01d5a-fc60-4aee-89be-50ecd3abb448" (UID: "38c01d5a-fc60-4aee-89be-50ecd3abb448"). InnerVolumeSpecName "kube-api-access-nb4jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.948383 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.948298 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl" pod "86b42ef8-935c-4a0d-b01e-848f3bc493bb" (UID: "86b42ef8-935c-4a0d-b01e-848f3bc493bb"). InnerVolumeSpecName "kube-api-access-gxghl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:21:59.948542 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.948498 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs" pod "86b42ef8-935c-4a0d-b01e-848f3bc493bb" (UID: "86b42ef8-935c-4a0d-b01e-848f3bc493bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:21:59.984409 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:21:59.980301 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bbf58b_bc2c_4e52_9bbb_5ced5699f15d.slice/crio-eb2d11f856efd2a4343466c5da172a2085226e256aec8b550dbfdfa2471e0998 WatchSource:0}: Error finding container eb2d11f856efd2a4343466c5da172a2085226e256aec8b550dbfdfa2471e0998: Status 404 returned error can't find the container with id eb2d11f856efd2a4343466c5da172a2085226e256aec8b550dbfdfa2471e0998 Nov 03 16:21:59.984409 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.983273 6319 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:21:59.994615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:21:59.994569 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4"] Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002333 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002367 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c01d5a-fc60-4aee-89be-50ecd3abb448-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002383 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/988c231f-9d8e-4873-8e01-8599e5d904d6-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002400 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b42ef8-935c-4a0d-b01e-848f3bc493bb-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002415 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4jp\" (UniqueName: \"kubernetes.io/projected/38c01d5a-fc60-4aee-89be-50ecd3abb448-kube-api-access-nb4jp\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002430 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b42ef8-935c-4a0d-b01e-848f3bc493bb-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.008339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.002444 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxghl\" (UniqueName: \"kubernetes.io/projected/86b42ef8-935c-4a0d-b01e-848f3bc493bb-kube-api-access-gxghl\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:00.066715 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.066677 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" event={"ID":"ab5e6618-836c-4f45-8ef3-7e9aaa552773","Type":"ContainerDied","Data":"b1b4581490c93fdab54283bfe3f5d020c0fd3bfc36c1d42f7be13816ec8cf121"} Nov 03 16:22:00.069048 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.066690 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj" Nov 03 16:22:00.070730 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.070664 6319 scope.go:122] "RemoveContainer" containerID="44b82474c2a9919f0ae441a09ec87f49277c742b983bccea582dbad4a5590a2b" Nov 03 16:22:00.079291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.077256 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" event={"ID":"def1b0ba-39a7-47ed-9dcd-bc84b2d14410","Type":"ContainerDied","Data":"0bfec1e6c12730854d32efe68851132774de1053a958082247d1f1994aa63cd0"} Nov 03 16:22:00.079291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.077390 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4" Nov 03 16:22:00.113451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.113158 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" event={"ID":"6d44abb0-b332-4714-9585-0be6d576d530","Type":"ContainerDied","Data":"b45e668d511fa16d196931e31e6786ef89fcaeb666ec45bb9f5233cb9669d750"} Nov 03 16:22:00.113451 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.113272 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt" Nov 03 16:22:00.163603 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.156322 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj"] Nov 03 16:22:00.189221 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.186355 6319 scope.go:122] "RemoveContainer" containerID="0f979ba2e4010ada07b3e85e833f2d740c58e3e95d56dd7ab428e59e1d55a38f" Nov 03 16:22:00.205385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.205274 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" event={"ID":"86b42ef8-935c-4a0d-b01e-848f3bc493bb","Type":"ContainerDied","Data":"1e2b524236a17d2a5f4a9f628f6916af771b976ac4687f41b89992d79326fef0"} Nov 03 16:22:00.205950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.205925 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5" Nov 03 16:22:00.294748 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.291440 6319 generic.go:356] "Generic (PLEG): container finished" podID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerID="00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296" exitCode=0 Nov 03 16:22:00.294748 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.291623 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" Nov 03 16:22:00.305398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.305360 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" event={"ID":"988c231f-9d8e-4873-8e01-8599e5d904d6","Type":"ContainerDied","Data":"00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296"} Nov 03 16:22:00.305969 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.305860 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2" event={"ID":"988c231f-9d8e-4873-8e01-8599e5d904d6","Type":"ContainerDied","Data":"f493ea990779128d52f8d401d676584be4dacf75656bfb3f13d4c9bdb1fe0419"} Nov 03 16:22:00.335565 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.335528 6319 generic.go:356] "Generic (PLEG): container finished" podID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerID="23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b" exitCode=0 Nov 03 16:22:00.335950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.335926 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" event={"ID":"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5","Type":"ContainerDied","Data":"23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b"} Nov 03 16:22:00.336327 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.336296 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" event={"ID":"4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5","Type":"ContainerDied","Data":"8f056af3fc6394aed0798554d7e1f792ba8599c47e99ab0e2a351ae6f38135ae"} Nov 03 16:22:00.337750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.337732 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m" Nov 03 16:22:00.340690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.340674 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" Nov 03 16:22:00.341148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.341127 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4" event={"ID":"11a0bf2d-0d7c-457c-9be4-b6e08db98c91","Type":"ContainerDied","Data":"0e2013989c264ec8a6ad8889af859b7a6ae6ea499a9e960461a002ecd71ff7c6"} Nov 03 16:22:00.346605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.346582 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" event={"ID":"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d","Type":"ContainerStarted","Data":"eb2d11f856efd2a4343466c5da172a2085226e256aec8b550dbfdfa2471e0998"} Nov 03 16:22:00.351750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.349563 6319 generic.go:356] "Generic (PLEG): container finished" podID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerID="f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245" exitCode=0 Nov 03 16:22:00.351750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.349646 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" event={"ID":"38c01d5a-fc60-4aee-89be-50ecd3abb448","Type":"ContainerDied","Data":"f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245"} Nov 03 16:22:00.351750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.349683 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" event={"ID":"38c01d5a-fc60-4aee-89be-50ecd3abb448","Type":"ContainerDied","Data":"c052215adbc65f69c5a395d9c20f417ec9b565ecf4e22565269c94e3fcb6bf68"} Nov 03 16:22:00.351750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.349765 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp" Nov 03 16:22:00.368061 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.366630 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" event={"ID":"c2451b7c-c27c-40b7-8e1f-ad322468b1d9","Type":"ContainerDied","Data":"fceab93a0807ce35be07fc32d9eb32a57b071a043beb71b3bfa4573a829b59b0"} Nov 03 16:22:00.368061 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.366753 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp" Nov 03 16:22:00.376149 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.376129 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-2j8zj"] Nov 03 16:22:00.459954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.459388 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" Nov 03 16:22:00.506765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.467883 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9" event={"ID":"1605541c-2e9c-42fd-9cd0-1606e0606e83","Type":"ContainerDied","Data":"68ec8f81e40a4ed63a583b3d0976d0f4e3136143d12437d1b35bc3844c6d2f47"} Nov 03 16:22:00.608956 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.596273 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4"] Nov 03 16:22:00.611199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.610269 6319 scope.go:122] "RemoveContainer" containerID="e4f9e6c52cdd2dc9e125fba77c77f1ef23a245c94a1b8e997fc27b6a897a1449" Nov 03 16:22:00.778755 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.778679 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" path="/var/lib/kubelet/pods/ab5e6618-836c-4f45-8ef3-7e9aaa552773/volumes" Nov 03 16:22:00.780284 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.780265 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d7196a-84566475d8-cfvp4"] Nov 03 16:22:00.780459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.780447 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt"] Nov 03 16:22:00.812076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.812044 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-8cfrt"] Nov 03 16:22:00.833007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.832963 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2"] Nov 03 16:22:00.843403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.842662 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:22:00.850285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.849291 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:22:00.855137 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.855061 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7196a-6f7f95c577-g4sd2"] Nov 03 16:22:00.860932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.860906 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp"] Nov 03 16:22:00.872629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.872133 6319 scope.go:122] "RemoveContainer" containerID="409710093bc6da7f4c532f14624bc119aeca75793ee9b63ff6fc8dfac84a4592" Nov 03 16:22:00.872629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.872262 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-ns9hp"] Nov 03 16:22:00.885277 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.885232 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m"] Nov 03 16:22:00.895420 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.895368 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-69cedd/distinct-2-w5-69cedd-kserve-559d65fcc8-whr4m"] Nov 03 16:22:00.917046 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.912199 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5"] Nov 03 16:22:00.933737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.920706 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-afb990/distinct-w6-afb990-kserve-78dfb494bd-jrfv5"] Nov 03 16:22:00.935736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.933948 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4"] Nov 03 16:22:00.935736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.934019 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d569fb-openshift-default-79b46bdf65-7pnt4"] Nov 03 16:22:00.935736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.935687 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp"] Nov 03 16:22:00.940084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.940064 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7196a-687d6496f5-vdxhp"] Nov 03 16:22:00.945055 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.945011 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9"] Nov 03 16:22:00.958552 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:00.958472 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c09b17/distinct-2-w1-c09b17-kserve-67df6c678c-nl5f9"] Nov 03 16:22:01.035863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.035174 6319 scope.go:122] "RemoveContainer" containerID="00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296" Nov 03 16:22:01.070300 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.070274 6319 scope.go:122] "RemoveContainer" containerID="00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296" Nov 03 16:22:01.071179 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:01.071010 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296\": container with ID starting with 00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296 not found: ID does not exist" containerID="00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296" Nov 03 16:22:01.071179 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.071051 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296"} err="failed to get container status \"00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296\": rpc error: code = NotFound desc = could not find container \"00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296\": container with ID starting with 00608c0365c155315e78adfa69553265612e434d5e5bebdc9a0a5f4258aa2296 not found: ID does not exist" Nov 03 16:22:01.071179 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.071078 6319 scope.go:122] "RemoveContainer" containerID="23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b" Nov 03 16:22:01.092142 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.092100 6319 scope.go:122] "RemoveContainer" containerID="23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b" Nov 03 16:22:01.096018 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:01.094405 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b\": container with ID starting with 23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b not found: ID does not exist" containerID="23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b" Nov 03 16:22:01.096018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.094447 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b"} err="failed to get container status \"23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b\": rpc error: code = NotFound desc = could not find container \"23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b\": container with ID starting with 23f180ba8e9b987e94e7de6a12f52d3228e69183cd7419bf8d0bc47d0e20183b not found: ID does not exist" Nov 03 16:22:01.096018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.094473 6319 scope.go:122] "RemoveContainer" containerID="6616e8c1e0f6148b01bc33f12fcb797bfc85249bb27fc6a5635e5820c1057d42" Nov 03 16:22:01.176427 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.176398 6319 scope.go:122] "RemoveContainer" containerID="f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245" Nov 03 16:22:01.194274 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.194246 6319 scope.go:122] "RemoveContainer" containerID="f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245" Nov 03 16:22:01.195715 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:01.195689 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245\": container with ID starting with f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245 not found: ID does not exist" containerID="f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245" Nov 03 16:22:01.195868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.195843 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245"} err="failed to get container status \"f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245\": rpc error: code = NotFound desc = could not find container \"f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245\": container with ID starting with f321a6ed8cad1e2d03b2731b939b91b3f77be3596acdb7113899f45776744245 not found: ID does not exist" Nov 03 16:22:01.195969 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.195960 6319 scope.go:122] "RemoveContainer" containerID="1b79ec7e98fc38638dbc8b291b3dd14fd6fd6c195134aaa812e073ea57019365" Nov 03 16:22:01.225032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.224973 6319 scope.go:122] "RemoveContainer" containerID="e29d73cbf9757619acc7e162052b77749fbb2019a2e9ecb3bfca1b028214430c" Nov 03 16:22:01.242086 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.242051 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:22:01.245017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.244959 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:22:01.365446 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.365365 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:22:01.368325 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.368302 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:22:01.509225 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.508738 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:01.509225 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:01.508803 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:02.261337 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.261301 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g"] Nov 03 16:22:02.262636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.262594 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" containerID="cri-o://af1d0ede3f52a84b68763dc1164e04821651f4b65fbd786164c2b61f1a7b78c7" gracePeriod=30 Nov 03 16:22:02.270663 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.270636 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh"] Nov 03 16:22:02.271274 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.271214 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" containerID="cri-o://7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29" gracePeriod=30 Nov 03 16:22:02.571487 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.570664 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:02.571487 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.570781 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:02.789047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.788190 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" path="/var/lib/kubelet/pods/11a0bf2d-0d7c-457c-9be4-b6e08db98c91/volumes" Nov 03 16:22:02.789047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.788940 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" path="/var/lib/kubelet/pods/1605541c-2e9c-42fd-9cd0-1606e0606e83/volumes" Nov 03 16:22:02.796010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.793763 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" path="/var/lib/kubelet/pods/38c01d5a-fc60-4aee-89be-50ecd3abb448/volumes" Nov 03 16:22:02.796010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.794461 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" path="/var/lib/kubelet/pods/4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5/volumes" Nov 03 16:22:02.796010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.795178 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d44abb0-b332-4714-9585-0be6d576d530" path="/var/lib/kubelet/pods/6d44abb0-b332-4714-9585-0be6d576d530/volumes" Nov 03 16:22:02.799017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.796668 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" path="/var/lib/kubelet/pods/86b42ef8-935c-4a0d-b01e-848f3bc493bb/volumes" Nov 03 16:22:02.799017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.797282 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" path="/var/lib/kubelet/pods/988c231f-9d8e-4873-8e01-8599e5d904d6/volumes" Nov 03 16:22:02.799017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.797965 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" path="/var/lib/kubelet/pods/c2451b7c-c27c-40b7-8e1f-ad322468b1d9/volumes" Nov 03 16:22:02.803014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:02.799886 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" path="/var/lib/kubelet/pods/def1b0ba-39a7-47ed-9dcd-bc84b2d14410/volumes" Nov 03 16:22:03.026605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:03.025428 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:22:03.029504 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:03.029424 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:22:03.656801 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:03.656762 6319 patch_prober.go:29] interesting pod/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:03.657351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:03.656831 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:06.518575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:06.518528 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf"] Nov 03 16:22:06.527286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:06.527259 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x"] Nov 03 16:22:06.646684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:06.646648 6319 generic.go:356] "Generic (PLEG): container finished" podID="34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" containerID="78d5edaedaa71236d4764425dcabf95eca32c0d9fccd20af904cb9843342f015" exitCode=0 Nov 03 16:22:06.646899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:06.646858 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" event={"ID":"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d","Type":"ContainerDied","Data":"78d5edaedaa71236d4764425dcabf95eca32c0d9fccd20af904cb9843342f015"} Nov 03 16:22:07.529335 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.529061 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:22:07.604502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604413 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604481 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604527 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604559 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604620 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604655 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604775 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.604903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604886 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.605055 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.604966 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d\") pod \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\" (UID: \"effc8f7a-b111-43ba-ae49-cc7698dfe3c2\") " Nov 03 16:22:07.606176 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.605758 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.606176 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.606038 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.606312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.606211 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.606312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.606275 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.606500 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.606479 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:07.612400 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.612351 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:07.612605 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.612577 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.613046 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.612962 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:07.614503 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.614444 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d" pod "effc8f7a-b111-43ba-ae49-cc7698dfe3c2" (UID: "effc8f7a-b111-43ba-ae49-cc7698dfe3c2"). InnerVolumeSpecName "kube-api-access-mxn4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:07.656207 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.656180 6319 generic.go:356] "Generic (PLEG): container finished" podID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerID="af1d0ede3f52a84b68763dc1164e04821651f4b65fbd786164c2b61f1a7b78c7" exitCode=0 Nov 03 16:22:07.656316 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.656247 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" event={"ID":"c19cf4b4-48e2-4276-9a4a-f76af7843f29","Type":"ContainerDied","Data":"af1d0ede3f52a84b68763dc1164e04821651f4b65fbd786164c2b61f1a7b78c7"} Nov 03 16:22:07.656381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.656315 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" event={"ID":"c19cf4b4-48e2-4276-9a4a-f76af7843f29","Type":"ContainerDied","Data":"3ac219395f855d991efdb4b9010ec2e82098cb1a21d56a33b8f54df15392a93c"} Nov 03 16:22:07.656381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.656339 6319 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac219395f855d991efdb4b9010ec2e82098cb1a21d56a33b8f54df15392a93c" Nov 03 16:22:07.656381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.656358 6319 scope.go:122] "RemoveContainer" containerID="e8bd12dbbb294d2ec0a9c5b4a8e518fe8cd2746d9cef9f3f23a041a570a2b794" Nov 03 16:22:07.658285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.658214 6319 generic.go:356] "Generic (PLEG): container finished" podID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerID="7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29" exitCode=0 Nov 03 16:22:07.658285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.658260 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" event={"ID":"effc8f7a-b111-43ba-ae49-cc7698dfe3c2","Type":"ContainerDied","Data":"7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29"} Nov 03 16:22:07.658285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.658292 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" Nov 03 16:22:07.658285 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.658321 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh" event={"ID":"effc8f7a-b111-43ba-ae49-cc7698dfe3c2","Type":"ContainerDied","Data":"a6452c937e5aee62e6f238b6de54b8fc01ced4b6ab004a4a9d5248606a0182bc"} Nov 03 16:22:07.677945 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.677917 6319 scope.go:122] "RemoveContainer" containerID="7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29" Nov 03 16:22:07.681205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.681183 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:22:07.689140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.689115 6319 scope.go:122] "RemoveContainer" containerID="7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29" Nov 03 16:22:07.691218 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:07.689450 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29\": container with ID starting with 7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29 not found: ID does not exist" containerID="7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29" Nov 03 16:22:07.691218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.689476 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29"} err="failed to get container status \"7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29\": rpc error: code = NotFound desc = could not find container \"7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29\": container with ID starting with 7bde2eb2dae7d4467816b4cbfc048f35d4296dc4d33e01bf5149e61c370d6f29 not found: ID does not exist" Nov 03 16:22:07.697966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.697926 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh"] Nov 03 16:22:07.705309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.705286 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-d2gsh"] Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708272 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708301 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708318 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708333 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708346 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708361 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxn4d\" (UniqueName: \"kubernetes.io/projected/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-kube-api-access-mxn4d\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708375 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708389 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.709020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.708402 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/effc8f7a-b111-43ba-ae49-cc7698dfe3c2-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.840557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840510 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.840751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840578 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.840751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840636 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.840863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840745 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.840863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840787 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.840863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840850 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.841028 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840883 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.841028 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.840920 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.841214 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.841194 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token\") pod \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\" (UID: \"c19cf4b4-48e2-4276-9a4a-f76af7843f29\") " Nov 03 16:22:07.842283 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.842253 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.842283 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.842270 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.842581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.842559 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:07.845920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.845632 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.845920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.845859 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.901578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.901337 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:07.903816 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.903746 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "kube-api-access-xzwln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:07.904323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.904278 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:07.904947 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.904922 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo" pod "c19cf4b4-48e2-4276-9a4a-f76af7843f29" (UID: "c19cf4b4-48e2-4276-9a4a-f76af7843f29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:07.916459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.916435 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943811 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943844 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943859 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943873 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943887 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwln\" (UniqueName: \"kubernetes.io/projected/c19cf4b4-48e2-4276-9a4a-f76af7843f29-kube-api-access-xzwln\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943900 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943913 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943928 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:07.943966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:07.943941 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19cf4b4-48e2-4276-9a4a-f76af7843f29-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:08.045211 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.045182 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca\") pod \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " Nov 03 16:22:08.045413 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.045245 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca\") pod \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " Nov 03 16:22:08.045495 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.045432 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw\" (UniqueName: \"kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw\") pod \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\" (UID: \"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d\") " Nov 03 16:22:08.045553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.045521 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca" pod "34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" (UID: "34bbf58b-bc2c-4e52-9bbb-5ced5699f15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:08.046082 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.046064 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-openshift-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:08.047611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.047586 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca" pod "34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" (UID: "34bbf58b-bc2c-4e52-9bbb-5ced5699f15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:08.047756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.047734 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw" pod "34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" (UID: "34bbf58b-bc2c-4e52-9bbb-5ced5699f15d"). InnerVolumeSpecName "kube-api-access-9h7fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:08.147867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.147792 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7fw\" (UniqueName: \"kubernetes.io/projected/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-kube-api-access-9h7fw\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:08.147867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.147822 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d-maas-api-xks-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:08.661576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.661544 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" path="/var/lib/kubelet/pods/effc8f7a-b111-43ba-ae49-cc7698dfe3c2/volumes" Nov 03 16:22:08.667225 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.667201 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" Nov 03 16:22:08.667464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.667202 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4" event={"ID":"34bbf58b-bc2c-4e52-9bbb-5ced5699f15d","Type":"ContainerDied","Data":"eb2d11f856efd2a4343466c5da172a2085226e256aec8b550dbfdfa2471e0998"} Nov 03 16:22:08.667593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.667495 6319 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2d11f856efd2a4343466c5da172a2085226e256aec8b550dbfdfa2471e0998" Nov 03 16:22:08.668961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.668939 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g" Nov 03 16:22:08.711187 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.711157 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g"] Nov 03 16:22:08.713867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:08.713841 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7196a-openshift-default-54c66d4895-xtl6g"] Nov 03 16:22:10.730268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:10.730230 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" path="/var/lib/kubelet/pods/c19cf4b4-48e2-4276-9a4a-f76af7843f29/volumes" Nov 03 16:22:12.209009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:12.203841 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v"] Nov 03 16:22:14.933203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:14.925654 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g"] Nov 03 16:22:14.933203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:14.925884 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" containerID="cri-o://3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9" gracePeriod=30 Nov 03 16:22:14.934259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:14.934218 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p"] Nov 03 16:22:14.934453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:14.934423 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" containerID="cri-o://ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f" gracePeriod=30 Nov 03 16:22:15.003723 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:15.003673 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd"] Nov 03 16:22:15.003971 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:15.003946 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" containerID="cri-o://e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11" gracePeriod=30 Nov 03 16:22:15.016180 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:15.016150 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp"] Nov 03 16:22:15.016358 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:15.016340 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" containerID="cri-o://a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a" gracePeriod=30 Nov 03 16:22:16.560580 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:16.560542 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 16:22:16.586197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:16.586154 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:17.163017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.155069 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:17.163017 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.155148 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:17.220842 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.220799 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:17.222312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.221127 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerName="payload-pre-processing" containerID="cri-o://7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4" gracePeriod=25 Nov 03 16:22:17.600138 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.582093 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:17.943235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.933095 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:22:17.943235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:17.942345 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:18.050064 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.045349 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:18.050064 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.045418 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:18.230092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.204602 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:18.231011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.230482 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 03 16:22:18.301298 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.230973 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:22:18.708212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.694600 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:22:18.766083 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.729464 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:22:18.815011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.789867 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:18.815011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.789946 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:18.924010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.903102 6319 patch_prober.go:29] interesting pod/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:18.924010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:18.903208 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:19.081065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.074558 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:19.081065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.074640 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:19.125112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.122933 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:19.125112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.123490 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:22:19.245611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.225049 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:19.245611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.225504 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:22:19.467003 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.466941 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:19.807842 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.807797 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:22:19.840961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.840908 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 16:22:19.852581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.852544 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:19.854592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.852859 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:19.869159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.869121 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:19.869457 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:19.869422 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.127032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.126927 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.143012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.140551 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:20.143012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.140667 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.315814 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.291414 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.316646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.316603 6319 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.447169 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.447278 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.447393 6319 patch_prober.go:29] interesting pod/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.447420 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.493601 6319 patch_prober.go:29] interesting pod/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.493684 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.554287 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.554338 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.610140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.604100 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:20.952895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:20.889422 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:21.373548 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.373343 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:22:21.607013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.600298 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 16:22:21.619374 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.619326 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:21.622567 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.622534 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" containerID="cri-o://7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5" gracePeriod=45 Nov 03 16:22:21.633088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.626490 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" containerID="cri-o://c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64" gracePeriod=45 Nov 03 16:22:21.722069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.722041 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:22:21.752087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.741502 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:22:21.752087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.748554 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:22:21.802393 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:21.802320 6319 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6251865_47a9_408e_a9eb_9546ec0e4175.slice/crio-conmon-3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f71d83_d648_4a63_9289_60904622b040.slice/crio-a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6251865_47a9_408e_a9eb_9546ec0e4175.slice/crio-41a614c33314e02309d67a9249b111d94dec5435f09460db27927773e5c041b8\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ecd076_c57d_4ba5_874c_70c829037b02.slice/crio-conmon-e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6251865_47a9_408e_a9eb_9546ec0e4175.slice/crio-3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f71d83_d648_4a63_9289_60904622b040.slice/crio-conmon-a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3595af_163a_4339_858f_4f1885ce04a2.slice/crio-conmon-ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ecd076_c57d_4ba5_874c_70c829037b02.slice/crio-e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3595af_163a_4339_858f_4f1885ce04a2.slice/crio-ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:22:21.810596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.810567 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr"] Nov 03 16:22:21.825124 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.819919 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8"] Nov 03 16:22:21.825124 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.820027 6319 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:22:21.866128 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.865146 6319 generic.go:356] "Generic (PLEG): container finished" podID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerID="3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9" exitCode=0 Nov 03 16:22:21.866128 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.865342 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" event={"ID":"b6251865-47a9-408e-a9eb-9546ec0e4175","Type":"ContainerDied","Data":"3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9"} Nov 03 16:22:21.871550 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.871210 6319 generic.go:356] "Generic (PLEG): container finished" podID="3d3595af-163a-4339-858f-4f1885ce04a2" containerID="ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f" exitCode=0 Nov 03 16:22:21.871550 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.871301 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" event={"ID":"3d3595af-163a-4339-858f-4f1885ce04a2","Type":"ContainerDied","Data":"ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f"} Nov 03 16:22:21.876205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.876178 6319 generic.go:356] "Generic (PLEG): container finished" podID="b0f71d83-d648-4a63-9289-60904622b040" containerID="a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a" exitCode=0 Nov 03 16:22:21.876580 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.876553 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" event={"ID":"b0f71d83-d648-4a63-9289-60904622b040","Type":"ContainerDied","Data":"a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a"} Nov 03 16:22:21.879123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.878662 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:22:21.894201 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:21.890272 6319 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9214762_f1a2_4155_b8a4_6b01cadc36da.slice/crio-conmon-7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9214762_f1a2_4155_b8a4_6b01cadc36da.slice/crio-7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:22:21.894201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.890537 6319 generic.go:356] "Generic (PLEG): container finished" podID="23ecd076-c57d-4ba5-874c-70c829037b02" containerID="e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11" exitCode=0 Nov 03 16:22:21.894201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:21.890575 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" event={"ID":"23ecd076-c57d-4ba5-874c-70c829037b02","Type":"ContainerDied","Data":"e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11"} Nov 03 16:22:22.218169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.213422 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:22:22.245680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.242078 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:22:22.345223 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345190 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345251 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.345429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345328 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345355 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.345429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345388 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345437 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345463 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345519 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345571 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345642 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345665 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345716 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345747 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.345811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345785 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.346436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345833 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.346436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345898 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.346436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345935 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy\") pod \"23ecd076-c57d-4ba5-874c-70c829037b02\" (UID: \"23ecd076-c57d-4ba5-874c-70c829037b02\") " Nov 03 16:22:22.346436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.345964 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy\") pod \"3d3595af-163a-4339-858f-4f1885ce04a2\" (UID: \"3d3595af-163a-4339-858f-4f1885ce04a2\") " Nov 03 16:22:22.347159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.347092 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.353065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.352733 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.353279 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.353220 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.353433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.353328 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:22.353508 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.353461 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.353568 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.353533 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.354492 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.354467 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.354721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.354683 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.354798 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.354767 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.355834 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.355653 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:22.355834 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.355711 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.357292 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.357233 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "kube-api-access-pbdpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.358108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.357999 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:22.358961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.358933 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token" pod "3d3595af-163a-4339-858f-4f1885ce04a2" (UID: "3d3595af-163a-4339-858f-4f1885ce04a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.359434 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.359410 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "kube-api-access-xdgb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.359661 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.359640 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:22.360727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.360707 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.364621 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.364592 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy" pod "23ecd076-c57d-4ba5-874c-70c829037b02" (UID: "23ecd076-c57d-4ba5-874c-70c829037b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.369752 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.369732 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:22:22.376275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.376249 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:22:22.396074 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.396048 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:22:22.400189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.400165 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:22:22.447719 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447628 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm\") pod \"c9214762-f1a2-4155-b8a4-6b01cadc36da\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " Nov 03 16:22:22.447719 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447688 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.447944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447742 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.447944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447769 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.447944 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447893 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache\") pod \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " Nov 03 16:22:22.448132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447949 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home\") pod \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " Nov 03 16:22:22.448132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.447978 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.448132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448039 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.448132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448070 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z\" (UniqueName: \"kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z\") pod \"c9214762-f1a2-4155-b8a4-6b01cadc36da\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " Nov 03 16:22:22.448132 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448113 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.448418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448125 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.448418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448198 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache" pod "9b81517c-8ea1-47ba-ac5d-f45983c8df96" (UID: "9b81517c-8ea1-47ba-ac5d-f45983c8df96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.448418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448222 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv\" (UniqueName: \"kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv\") pod \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " Nov 03 16:22:22.448418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448274 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.448418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448345 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.448418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448385 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448447 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448486 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs\") pod \"c9214762-f1a2-4155-b8a4-6b01cadc36da\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448514 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs\") pod \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448577 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448582 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir\") pod \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448637 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448681 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache\") pod \"c9214762-f1a2-4155-b8a4-6b01cadc36da\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " Nov 03 16:22:22.448890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448794 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir" pod "9b81517c-8ea1-47ba-ac5d-f45983c8df96" (UID: "9b81517c-8ea1-47ba-ac5d-f45983c8df96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.449424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448960 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.449424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.448973 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.449424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.449191 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.449575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.449433 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.449575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.449478 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.450706 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.450683 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache" pod "c9214762-f1a2-4155-b8a4-6b01cadc36da" (UID: "c9214762-f1a2-4155-b8a4-6b01cadc36da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.451464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.451056 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.451464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.451062 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home" pod "9b81517c-8ea1-47ba-ac5d-f45983c8df96" (UID: "9b81517c-8ea1-47ba-ac5d-f45983c8df96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.451464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.451162 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm" pod "c9214762-f1a2-4155-b8a4-6b01cadc36da" (UID: "c9214762-f1a2-4155-b8a4-6b01cadc36da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.452154 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.451539 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.452529 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452239 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir\") pod \"c9214762-f1a2-4155-b8a4-6b01cadc36da\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " Nov 03 16:22:22.452529 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452291 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.452767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452531 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.452767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452591 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert\") pod \"b6251865-47a9-408e-a9eb-9546ec0e4175\" (UID: \"b6251865-47a9-408e-a9eb-9546ec0e4175\") " Nov 03 16:22:22.452767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452632 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm\") pod \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\" (UID: \"9b81517c-8ea1-47ba-ac5d-f45983c8df96\") " Nov 03 16:22:22.452767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452690 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.452767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452724 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home\") pod \"c9214762-f1a2-4155-b8a4-6b01cadc36da\" (UID: \"c9214762-f1a2-4155-b8a4-6b01cadc36da\") " Nov 03 16:22:22.453053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452779 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.453053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452815 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.453053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.452845 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy\") pod \"b0f71d83-d648-4a63-9289-60904622b040\" (UID: \"b0f71d83-d648-4a63-9289-60904622b040\") " Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453808 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453834 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453849 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453863 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453877 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453890 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453903 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453921 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453936 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbdpw\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-kube-api-access-pbdpw\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453948 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453961 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.453974 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454007 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454022 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3595af-163a-4339-858f-4f1885ce04a2-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454036 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454057 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454071 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6251865-47a9-408e-a9eb-9546ec0e4175-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454085 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454099 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3595af-163a-4339-858f-4f1885ce04a2-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454112 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454130 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454146 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454164 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454190 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454205 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ecd076-c57d-4ba5-874c-70c829037b02-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454218 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3595af-163a-4339-858f-4f1885ce04a2-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454232 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ecd076-c57d-4ba5-874c-70c829037b02-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454247 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3595af-163a-4339-858f-4f1885ce04a2-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454260 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgb9\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-kube-api-access-xdgb9\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454274 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ecd076-c57d-4ba5-874c-70c829037b02-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454287 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454299 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ecd076-c57d-4ba5-874c-70c829037b02-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.454543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.454305 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir" pod "c9214762-f1a2-4155-b8a4-6b01cadc36da" (UID: "c9214762-f1a2-4155-b8a4-6b01cadc36da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.456389 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.455611 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home" pod "c9214762-f1a2-4155-b8a4-6b01cadc36da" (UID: "c9214762-f1a2-4155-b8a4-6b01cadc36da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.456956 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.456644 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv" pod "9b81517c-8ea1-47ba-ac5d-f45983c8df96" (UID: "9b81517c-8ea1-47ba-ac5d-f45983c8df96"). InnerVolumeSpecName "kube-api-access-2nllv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.456956 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.456713 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs" pod "9b81517c-8ea1-47ba-ac5d-f45983c8df96" (UID: "9b81517c-8ea1-47ba-ac5d-f45983c8df96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:22.456956 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.456759 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs" pod "c9214762-f1a2-4155-b8a4-6b01cadc36da" (UID: "c9214762-f1a2-4155-b8a4-6b01cadc36da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:22.457474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.457363 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:22.458036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.457785 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.458036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.457939 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:22.460632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.460278 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "kube-api-access-cx968". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.463126 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.463089 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.464202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.464181 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.465178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.465155 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "kube-api-access-zdlk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.465178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.465164 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo" pod "b0f71d83-d648-4a63-9289-60904622b040" (UID: "b0f71d83-d648-4a63-9289-60904622b040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:22.466698 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.466678 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm" pod "9b81517c-8ea1-47ba-ac5d-f45983c8df96" (UID: "9b81517c-8ea1-47ba-ac5d-f45983c8df96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:22.466793 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.466695 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z" pod "c9214762-f1a2-4155-b8a4-6b01cadc36da" (UID: "c9214762-f1a2-4155-b8a4-6b01cadc36da"). InnerVolumeSpecName "kube-api-access-w488z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:22.469166 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.469140 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo" pod "b6251865-47a9-408e-a9eb-9546ec0e4175" (UID: "b6251865-47a9-408e-a9eb-9546ec0e4175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:22.556259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556228 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w488z\" (UniqueName: \"kubernetes.io/projected/c9214762-f1a2-4155-b8a4-6b01cadc36da-kube-api-access-w488z\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556254 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nllv\" (UniqueName: \"kubernetes.io/projected/9b81517c-8ea1-47ba-ac5d-f45983c8df96-kube-api-access-2nllv\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556259 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556265 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0f71d83-d648-4a63-9289-60904622b040-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556275 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9214762-f1a2-4155-b8a4-6b01cadc36da-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556284 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b81517c-8ea1-47ba-ac5d-f45983c8df96-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556292 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdlk9\" (UniqueName: \"kubernetes.io/projected/b6251865-47a9-408e-a9eb-9546ec0e4175-kube-api-access-zdlk9\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556300 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556308 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6251865-47a9-408e-a9eb-9546ec0e4175-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556316 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556323 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6251865-47a9-408e-a9eb-9546ec0e4175-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556331 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b81517c-8ea1-47ba-ac5d-f45983c8df96-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556339 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx968\" (UniqueName: \"kubernetes.io/projected/b0f71d83-d648-4a63-9289-60904622b040-kube-api-access-cx968\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556347 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9214762-f1a2-4155-b8a4-6b01cadc36da-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556356 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0f71d83-d648-4a63-9289-60904622b040-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556367 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.556563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.556378 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0f71d83-d648-4a63-9289-60904622b040-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:22.898677 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.898643 6319 generic.go:356] "Generic (PLEG): container finished" podID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerID="c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64" exitCode=0 Nov 03 16:22:22.898831 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.898702 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" Nov 03 16:22:22.898831 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.898723 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" event={"ID":"9b81517c-8ea1-47ba-ac5d-f45983c8df96","Type":"ContainerDied","Data":"c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64"} Nov 03 16:22:22.898831 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.898794 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x" event={"ID":"9b81517c-8ea1-47ba-ac5d-f45983c8df96","Type":"ContainerDied","Data":"82718e1333d969fec8ae34d790ab35be1a7dbf4713e461ba5725c47188b4e45c"} Nov 03 16:22:22.898831 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.898819 6319 scope.go:122] "RemoveContainer" containerID="c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64" Nov 03 16:22:22.900413 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.900383 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" Nov 03 16:22:22.900563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.900447 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd" event={"ID":"23ecd076-c57d-4ba5-874c-70c829037b02","Type":"ContainerDied","Data":"e47cda84a9ff26225ba7b05f1fbad54f1863af78aef40e9c3d4eb6966dc8920c"} Nov 03 16:22:22.902438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.902403 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" event={"ID":"b6251865-47a9-408e-a9eb-9546ec0e4175","Type":"ContainerDied","Data":"41a614c33314e02309d67a9249b111d94dec5435f09460db27927773e5c041b8"} Nov 03 16:22:22.902577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.902441 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g" Nov 03 16:22:22.904339 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.904317 6319 generic.go:356] "Generic (PLEG): container finished" podID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerID="7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5" exitCode=0 Nov 03 16:22:22.904438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.904374 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" Nov 03 16:22:22.904438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.904407 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" event={"ID":"c9214762-f1a2-4155-b8a4-6b01cadc36da","Type":"ContainerDied","Data":"7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5"} Nov 03 16:22:22.904438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.904431 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf" event={"ID":"c9214762-f1a2-4155-b8a4-6b01cadc36da","Type":"ContainerDied","Data":"58b956e94264552edceb815a93a140aacb0ffe93fcef252a4942df0323a0a074"} Nov 03 16:22:22.910033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.905960 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" Nov 03 16:22:22.910033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.905995 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p" event={"ID":"3d3595af-163a-4339-858f-4f1885ce04a2","Type":"ContainerDied","Data":"cbe91d70ad3ea93f8023a5cd9a3a31f6746495bc7dd4ba1e95f3feec87e49649"} Nov 03 16:22:22.910033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.907911 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" event={"ID":"b0f71d83-d648-4a63-9289-60904622b040","Type":"ContainerDied","Data":"4b15ce8fc163ae70cc0dd94479727bfaed194ad420ab07afbf36a5882e3bdf60"} Nov 03 16:22:22.910033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.907958 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp" Nov 03 16:22:22.911805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.911785 6319 scope.go:122] "RemoveContainer" containerID="c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64" Nov 03 16:22:22.912141 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:22.912121 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64\": container with ID starting with c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64 not found: ID does not exist" containerID="c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64" Nov 03 16:22:22.912236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.912151 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64"} err="failed to get container status \"c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64\": rpc error: code = NotFound desc = could not find container \"c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64\": container with ID starting with c54d6d6b31d19c6db22c574fbbdb0f462428a27d018839e8e2d408800dd2cb64 not found: ID does not exist" Nov 03 16:22:22.912236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.912174 6319 scope.go:122] "RemoveContainer" containerID="e41b2eb1654f9da4a48fc945bf574e5667c8dc54e6bf6a444c4bfe8f8790ae11" Nov 03 16:22:22.920904 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.920882 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd"] Nov 03 16:22:22.923161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.923144 6319 scope.go:122] "RemoveContainer" containerID="3aa9bafa527051cb7219bcbae7fbcbd0ac666f60c955a57aa456c521606e9ec9" Nov 03 16:22:22.929775 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.929755 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-tx9vd"] Nov 03 16:22:22.933652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.933635 6319 scope.go:122] "RemoveContainer" containerID="7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5" Nov 03 16:22:22.944652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.944634 6319 scope.go:122] "RemoveContainer" containerID="7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5" Nov 03 16:22:22.945053 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:22.945026 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5\": container with ID starting with 7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5 not found: ID does not exist" containerID="7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5" Nov 03 16:22:22.945148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.945062 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5"} err="failed to get container status \"7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5\": rpc error: code = NotFound desc = could not find container \"7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5\": container with ID starting with 7a4af34dd3fa0f0a53db7cbcdca38692331198f16fef11328d29e3cea711e5c5 not found: ID does not exist" Nov 03 16:22:22.945148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.945085 6319 scope.go:122] "RemoveContainer" containerID="ceb81a3fcf07a801e3750488151ee1f6217324956ed8154fb135269a0437122f" Nov 03 16:22:22.955015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.954834 6319 scope.go:122] "RemoveContainer" containerID="a3eacfddde5c604e26732fcf059ada3c357cbf3e6bab6b618c40f88e6aff9b4a" Nov 03 16:22:22.988466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:22.988404 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp"] Nov 03 16:22:23.006268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.006246 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-xdtzp"] Nov 03 16:22:23.036603 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.036583 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p"] Nov 03 16:22:23.053125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.053105 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-lzl5p"] Nov 03 16:22:23.064996 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.064964 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x"] Nov 03 16:22:23.068468 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.068450 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-w4-4a19cd-kserve-76c4b5456c-thn2x"] Nov 03 16:22:23.082112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.082087 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g"] Nov 03 16:22:23.090351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.090331 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c09b17-openshift-default-cc77495ff-6vd7g"] Nov 03 16:22:23.112192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.112169 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf"] Nov 03 16:22:23.118749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:23.118718 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4a19cd/distinct-2-w4-4a19cd-kserve-6d8c8f444-vnxsf"] Nov 03 16:22:24.643734 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:24.643706 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" path="/var/lib/kubelet/pods/23ecd076-c57d-4ba5-874c-70c829037b02/volumes" Nov 03 16:22:24.644271 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:24.644257 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" path="/var/lib/kubelet/pods/3d3595af-163a-4339-858f-4f1885ce04a2/volumes" Nov 03 16:22:24.644729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:24.644684 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" path="/var/lib/kubelet/pods/9b81517c-8ea1-47ba-ac5d-f45983c8df96/volumes" Nov 03 16:22:24.645541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:24.645519 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f71d83-d648-4a63-9289-60904622b040" path="/var/lib/kubelet/pods/b0f71d83-d648-4a63-9289-60904622b040/volumes" Nov 03 16:22:24.645998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:24.645969 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" path="/var/lib/kubelet/pods/b6251865-47a9-408e-a9eb-9546ec0e4175/volumes" Nov 03 16:22:24.646398 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:24.646385 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" path="/var/lib/kubelet/pods/c9214762-f1a2-4155-b8a4-6b01cadc36da/volumes" Nov 03 16:22:25.303036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.303005 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl"] Nov 03 16:22:25.303275 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.303249 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" containerID="cri-o://ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955" gracePeriod=30 Nov 03 16:22:25.328479 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.328445 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q"] Nov 03 16:22:25.328701 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.328679 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" containerID="cri-o://13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca" gracePeriod=30 Nov 03 16:22:25.334114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.334092 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l"] Nov 03 16:22:25.334312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.334291 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" containerID="cri-o://d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8" gracePeriod=30 Nov 03 16:22:25.336848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.336826 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff"] Nov 03 16:22:25.337114 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:25.337047 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" containerID="cri-o://a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3" gracePeriod=30 Nov 03 16:22:27.773509 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:27.773432 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:22:28.812263 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.812230 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p"] Nov 03 16:22:28.812712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.812454 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" containerID="cri-o://30aad3b85fdcb959db4130db50a5dc869210c6dc097603738a7197e470d7252a" gracePeriod=30 Nov 03 16:22:28.829921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.829883 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw"] Nov 03 16:22:28.830141 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.830116 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" containerID="cri-o://701bbbc36c3c37f4c094f9f9d5025a6c9c7d3457114f292d95154be14a09b798" gracePeriod=30 Nov 03 16:22:28.840532 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.840499 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9"] Nov 03 16:22:28.840805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.840774 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" containerID="cri-o://30f912c888e4c9137fbd52e60908ed20603e0d678ec3955de3a7fc2a5d3e7527" gracePeriod=30 Nov 03 16:22:28.994329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.976881 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg"] Nov 03 16:22:28.994329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:28.977117 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" podUID="5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" containerName="maas-api" containerID="cri-o://5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380" gracePeriod=30 Nov 03 16:22:29.024208 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.021912 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:22:29.481821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.481793 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:22:29.530467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.530437 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs\") pod \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " Nov 03 16:22:29.530622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.530521 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg\" (UniqueName: \"kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg\") pod \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " Nov 03 16:22:29.530622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.530555 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls\") pod \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " Nov 03 16:22:29.530622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.530615 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca\") pod \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\" (UID: \"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4\") " Nov 03 16:22:29.531341 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.531319 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca" pod "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" (UID: "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:29.534217 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.534187 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs" pod "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" (UID: "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:29.534780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.534748 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls" pod "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" (UID: "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:29.535109 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.535087 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg" pod "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" (UID: "5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4"). InnerVolumeSpecName "kube-api-access-82wtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:29.651814 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.645122 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:29.651814 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.645157 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wtg\" (UniqueName: \"kubernetes.io/projected/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-kube-api-access-82wtg\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:29.651814 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.645172 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:29.651814 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:29.645185 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.048063 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.048020 6319 generic.go:356] "Generic (PLEG): container finished" podID="5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" containerID="5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380" exitCode=0 Nov 03 16:22:30.048592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.048074 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" Nov 03 16:22:30.048592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.048113 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" event={"ID":"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4","Type":"ContainerDied","Data":"5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380"} Nov 03 16:22:30.048592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.048169 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg" event={"ID":"5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4","Type":"ContainerDied","Data":"97605590d29de15c667170fcdd7b02b74a45a82be1b7c20f5c9a6b9d45a552b6"} Nov 03 16:22:30.048592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.048197 6319 scope.go:122] "RemoveContainer" containerID="5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380" Nov 03 16:22:30.060862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.060839 6319 scope.go:122] "RemoveContainer" containerID="5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380" Nov 03 16:22:30.061155 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:30.061137 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380\": container with ID starting with 5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380 not found: ID does not exist" containerID="5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380" Nov 03 16:22:30.061221 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.061165 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380"} err="failed to get container status \"5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380\": rpc error: code = NotFound desc = could not find container \"5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380\": container with ID starting with 5fc28ffcad280f1805f269e643c19d7ffca0ed8ed5629b33a773e5f5e8c89380 not found: ID does not exist" Nov 03 16:22:30.080734 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.080704 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg"] Nov 03 16:22:30.086832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.086810 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4a19cd-6445d79fbc-c78jg"] Nov 03 16:22:30.654430 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.654371 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" path="/var/lib/kubelet/pods/5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4/volumes" Nov 03 16:22:30.693021 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.688235 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:22:30.719836 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.719588 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:22:30.724347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.724322 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:22:30.737930 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.737909 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:22:30.761880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.761830 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762006 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762046 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762190 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762252 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762304 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762280 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762378 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762422 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.762471 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.762457 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo\") pod \"46270e98-843d-4adc-9950-4bcf08c9f436\" (UID: \"46270e98-843d-4adc-9950-4bcf08c9f436\") " Nov 03 16:22:30.763722 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.763698 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.764081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.764059 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:30.764331 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.764312 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.765036 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.764972 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.765551 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.765520 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.767449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.767425 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.767544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.767499 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.767692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.767671 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:30.767757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.767715 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk" pod "46270e98-843d-4adc-9950-4bcf08c9f436" (UID: "46270e98-843d-4adc-9950-4bcf08c9f436"). InnerVolumeSpecName "kube-api-access-m6qkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.863781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.863754 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.863963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.863792 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.863963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.863841 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.863963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.863860 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.863963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.863904 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.863963 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.863941 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864020 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864059 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864089 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864092 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864127 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864186 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864211 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864214 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864222 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864266 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864324 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864355 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864358 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864389 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864403 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864435 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864437 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864471 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864486 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864543 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864579 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864601 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.864593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864611 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864655 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864697 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token\") pod \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\" (UID: \"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864727 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864752 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864787 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket\") pod \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\" (UID: \"c54ad8dd-f076-4c92-abe2-6366a3e7d803\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864814 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert\") pod \"fe8316fc-748b-4d59-9075-7bb4be758b7c\" (UID: \"fe8316fc-748b-4d59-9075-7bb4be758b7c\") " Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.864885 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865045 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865178 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865573 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865593 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865599 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865626 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865654 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865668 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865728 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865746 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865747 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865761 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865777 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865802 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865817 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46270e98-843d-4adc-9950-4bcf08c9f436-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865832 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865846 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865863 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865881 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865897 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865912 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.865913 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865928 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46270e98-843d-4adc-9950-4bcf08c9f436-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.867584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865945 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6qkk\" (UniqueName: \"kubernetes.io/projected/46270e98-843d-4adc-9950-4bcf08c9f436-kube-api-access-m6qkk\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.867584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.865962 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46270e98-843d-4adc-9950-4bcf08c9f436-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.867584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.866264 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:30.867584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.866487 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.867584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.867287 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "kube-api-access-55wvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.867584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.867543 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.868515 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.868469 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:30.868927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.868896 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.868927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.868878 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "kube-api-access-q5ztv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.869152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.869107 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:30.869390 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.869368 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.869477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.869453 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.869527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.869481 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token" pod "c54ad8dd-f076-4c92-abe2-6366a3e7d803" (UID: "c54ad8dd-f076-4c92-abe2-6366a3e7d803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.869845 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.869819 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:30.869937 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.869919 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv" pod "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" (UID: "1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1"). InnerVolumeSpecName "kube-api-access-b5brv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:30.870311 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.870294 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy" pod "fe8316fc-748b-4d59-9075-7bb4be758b7c" (UID: "fe8316fc-748b-4d59-9075-7bb4be758b7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:30.966868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966837 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.966868 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966865 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966875 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966884 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8316fc-748b-4d59-9075-7bb4be758b7c-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966895 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966903 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ztv\" (UniqueName: \"kubernetes.io/projected/fe8316fc-748b-4d59-9075-7bb4be758b7c-kube-api-access-q5ztv\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966913 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966921 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966930 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55wvk\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-kube-api-access-55wvk\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966938 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966946 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54ad8dd-f076-4c92-abe2-6366a3e7d803-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966953 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966961 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5brv\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-kube-api-access-b5brv\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966970 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8316fc-748b-4d59-9075-7bb4be758b7c-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.966977 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8316fc-748b-4d59-9075-7bb4be758b7c-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:30.967108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:30.967014 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:31.056917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.056882 6319 generic.go:356] "Generic (PLEG): container finished" podID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerID="a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3" exitCode=0 Nov 03 16:22:31.057466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.056927 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" Nov 03 16:22:31.057466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.056975 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" event={"ID":"fe8316fc-748b-4d59-9075-7bb4be758b7c","Type":"ContainerDied","Data":"a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3"} Nov 03 16:22:31.057466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.057046 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff" event={"ID":"fe8316fc-748b-4d59-9075-7bb4be758b7c","Type":"ContainerDied","Data":"f80f158cc79592e6e905c21cf8f8110fb8cbaa457dd20bdd91c201a54106c607"} Nov 03 16:22:31.057466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.057070 6319 scope.go:122] "RemoveContainer" containerID="a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3" Nov 03 16:22:31.058447 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.058421 6319 generic.go:356] "Generic (PLEG): container finished" podID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerID="13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca" exitCode=0 Nov 03 16:22:31.058580 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.058486 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" Nov 03 16:22:31.058580 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.058514 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" event={"ID":"c54ad8dd-f076-4c92-abe2-6366a3e7d803","Type":"ContainerDied","Data":"13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca"} Nov 03 16:22:31.058580 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.058571 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q" event={"ID":"c54ad8dd-f076-4c92-abe2-6366a3e7d803","Type":"ContainerDied","Data":"dfc5757037ebd7df88ee9b42ae3428c901d25465feee2923f6a10d770ffc5495"} Nov 03 16:22:31.060360 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.060337 6319 generic.go:356] "Generic (PLEG): container finished" podID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerID="d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8" exitCode=0 Nov 03 16:22:31.060478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.060389 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" Nov 03 16:22:31.060478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.060396 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" event={"ID":"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1","Type":"ContainerDied","Data":"d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8"} Nov 03 16:22:31.060478 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.060431 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l" event={"ID":"1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1","Type":"ContainerDied","Data":"ff537b99bc0458b7352ee256d2443f27b49768c54adeeb7b72d70ffe2d16a84d"} Nov 03 16:22:31.061812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.061793 6319 generic.go:356] "Generic (PLEG): container finished" podID="46270e98-843d-4adc-9950-4bcf08c9f436" containerID="ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955" exitCode=0 Nov 03 16:22:31.061907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.061834 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" Nov 03 16:22:31.061907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.061879 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" event={"ID":"46270e98-843d-4adc-9950-4bcf08c9f436","Type":"ContainerDied","Data":"ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955"} Nov 03 16:22:31.062020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.061917 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl" event={"ID":"46270e98-843d-4adc-9950-4bcf08c9f436","Type":"ContainerDied","Data":"03ae4397616a17e40908be55ab24ab75fc5f2c5243c8cdd528a1734b76371b33"} Nov 03 16:22:31.070661 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.070636 6319 scope.go:122] "RemoveContainer" containerID="a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3" Nov 03 16:22:31.073008 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:31.072966 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3\": container with ID starting with a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3 not found: ID does not exist" containerID="a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3" Nov 03 16:22:31.073096 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.073016 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3"} err="failed to get container status \"a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3\": rpc error: code = NotFound desc = could not find container \"a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3\": container with ID starting with a4660451a6a5ab93f9e397812b3a433269b16f3d051d276e6b2b67a4192924f3 not found: ID does not exist" Nov 03 16:22:31.073096 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.073040 6319 scope.go:122] "RemoveContainer" containerID="13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca" Nov 03 16:22:31.085211 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.085190 6319 scope.go:122] "RemoveContainer" containerID="13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca" Nov 03 16:22:31.085541 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:31.085511 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca\": container with ID starting with 13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca not found: ID does not exist" containerID="13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca" Nov 03 16:22:31.085646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.085552 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca"} err="failed to get container status \"13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca\": rpc error: code = NotFound desc = could not find container \"13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca\": container with ID starting with 13fff3fd55477aac3fb259294c2355cd702146f55c88014ae10b005c3e6406ca not found: ID does not exist" Nov 03 16:22:31.085646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.085574 6319 scope.go:122] "RemoveContainer" containerID="d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8" Nov 03 16:22:31.098611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.098585 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l"] Nov 03 16:22:31.098684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.098614 6319 scope.go:122] "RemoveContainer" containerID="191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e" Nov 03 16:22:31.106821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.106799 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-wvg6l"] Nov 03 16:22:31.110485 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.110464 6319 scope.go:122] "RemoveContainer" containerID="d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8" Nov 03 16:22:31.110749 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:31.110730 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8\": container with ID starting with d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8 not found: ID does not exist" containerID="d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8" Nov 03 16:22:31.110796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.110761 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8"} err="failed to get container status \"d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8\": rpc error: code = NotFound desc = could not find container \"d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8\": container with ID starting with d9dc3290e12f7ee597e5d10b26c554c349322ca239cacddc39f58d41c76496e8 not found: ID does not exist" Nov 03 16:22:31.110796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.110781 6319 scope.go:122] "RemoveContainer" containerID="191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e" Nov 03 16:22:31.111068 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:31.111048 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e\": container with ID starting with 191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e not found: ID does not exist" containerID="191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e" Nov 03 16:22:31.111117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.111073 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e"} err="failed to get container status \"191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e\": rpc error: code = NotFound desc = could not find container \"191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e\": container with ID starting with 191a91f6234d3947ccf710342135451744d2f660b1ce4d7c9757c2205500bf4e not found: ID does not exist" Nov 03 16:22:31.111117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.111087 6319 scope.go:122] "RemoveContainer" containerID="ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955" Nov 03 16:22:31.120186 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.120168 6319 scope.go:122] "RemoveContainer" containerID="ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955" Nov 03 16:22:31.120490 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:31.120470 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955\": container with ID starting with ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955 not found: ID does not exist" containerID="ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955" Nov 03 16:22:31.120559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.120496 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955"} err="failed to get container status \"ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955\": rpc error: code = NotFound desc = could not find container \"ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955\": container with ID starting with ece3d19f8ae8dc5b77ebbd9d0f44ff7f4a35e3f2cc92c9044b611d431a4d3955 not found: ID does not exist" Nov 03 16:22:31.121940 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.121915 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q"] Nov 03 16:22:31.128184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.128158 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7xn5q"] Nov 03 16:22:31.143221 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.143199 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff"] Nov 03 16:22:31.149688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.149665 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-sfcff"] Nov 03 16:22:31.166461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.166434 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl"] Nov 03 16:22:31.171375 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.171345 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-69cedd-openshift-default-796b97cd49-7dsvl"] Nov 03 16:22:31.697179 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.697098 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:31.697179 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:31.697166 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:32.095890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:32.095860 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:32.096288 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:32.095912 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:32.643550 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:32.643519 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" path="/var/lib/kubelet/pods/1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1/volumes" Nov 03 16:22:32.643998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:32.643969 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" path="/var/lib/kubelet/pods/46270e98-843d-4adc-9950-4bcf08c9f436/volumes" Nov 03 16:22:32.644467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:32.644452 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" path="/var/lib/kubelet/pods/c54ad8dd-f076-4c92-abe2-6366a3e7d803/volumes" Nov 03 16:22:32.645205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:32.645191 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" path="/var/lib/kubelet/pods/fe8316fc-748b-4d59-9075-7bb4be758b7c/volumes" Nov 03 16:22:34.089446 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.089351 6319 generic.go:356] "Generic (PLEG): container finished" podID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerID="30f912c888e4c9137fbd52e60908ed20603e0d678ec3955de3a7fc2a5d3e7527" exitCode=0 Nov 03 16:22:34.089895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.089473 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" event={"ID":"1eb937f4-88f8-40c9-bc40-615c9b81931a","Type":"ContainerDied","Data":"30f912c888e4c9137fbd52e60908ed20603e0d678ec3955de3a7fc2a5d3e7527"} Nov 03 16:22:34.091574 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.091537 6319 generic.go:356] "Generic (PLEG): container finished" podID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerID="30aad3b85fdcb959db4130db50a5dc869210c6dc097603738a7197e470d7252a" exitCode=0 Nov 03 16:22:34.091750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.091728 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" event={"ID":"6115cfdb-b425-4cff-a4ee-82e5452f98f3","Type":"ContainerDied","Data":"30aad3b85fdcb959db4130db50a5dc869210c6dc097603738a7197e470d7252a"} Nov 03 16:22:34.093890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.093859 6319 generic.go:356] "Generic (PLEG): container finished" podID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerID="701bbbc36c3c37f4c094f9f9d5025a6c9c7d3457114f292d95154be14a09b798" exitCode=0 Nov 03 16:22:34.094029 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.093950 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" event={"ID":"c432fb68-4436-42fe-86cc-eaddc04711d1","Type":"ContainerDied","Data":"701bbbc36c3c37f4c094f9f9d5025a6c9c7d3457114f292d95154be14a09b798"} Nov 03 16:22:34.127362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.127334 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:22:34.161533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.161504 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:22:34.179876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.179853 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:22:34.206327 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206295 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206343 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206372 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206553 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206510 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206563 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206624 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.206716 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206686 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206792 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.206871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206816 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.206871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.206830 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.207115 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.207089 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.207115 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.207101 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:34.207244 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.207225 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs\") pod \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\" (UID: \"6115cfdb-b425-4cff-a4ee-82e5452f98f3\") " Nov 03 16:22:34.208193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.208119 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.208302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.208215 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.208461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.208398 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.208461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.208424 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.208461 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.208438 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.209395 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.209364 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.209591 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.209558 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "kube-api-access-6z2jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:34.209781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.209758 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:34.212331 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.212306 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo" pod "6115cfdb-b425-4cff-a4ee-82e5452f98f3" (UID: "6115cfdb-b425-4cff-a4ee-82e5452f98f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:34.309137 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309103 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309155 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309187 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309215 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309286 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309323 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309356 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309385 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309381 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309433 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309501 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309533 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309607 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309631 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309714 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309746 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309781 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309824 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket\") pod \"1eb937f4-88f8-40c9-bc40-615c9b81931a\" (UID: \"1eb937f4-88f8-40c9-bc40-615c9b81931a\") " Nov 03 16:22:34.309866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309854 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket\") pod \"c432fb68-4436-42fe-86cc-eaddc04711d1\" (UID: \"c432fb68-4436-42fe-86cc-eaddc04711d1\") " Nov 03 16:22:34.310433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309498 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.310433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309629 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.310433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309749 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.310433 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.309946 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.310643 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.310488 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.310917 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311260 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311279 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311298 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311326 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311346 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311363 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb937f4-88f8-40c9-bc40-615c9b81931a-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311380 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311401 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311418 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z2jk\" (UniqueName: \"kubernetes.io/projected/6115cfdb-b425-4cff-a4ee-82e5452f98f3-kube-api-access-6z2jk\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311434 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6115cfdb-b425-4cff-a4ee-82e5452f98f3-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.311798 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312008 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312028 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312074 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312279 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312300 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:34.312648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312487 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "kube-api-access-wgw76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:34.313559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.312662 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:34.320803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.320774 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token" pod "1eb937f4-88f8-40c9-bc40-615c9b81931a" (UID: "1eb937f4-88f8-40c9-bc40-615c9b81931a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:34.320939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.320851 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:34.321021 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.320970 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "kube-api-access-k866v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:34.321779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.321758 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo" pod "c432fb68-4436-42fe-86cc-eaddc04711d1" (UID: "c432fb68-4436-42fe-86cc-eaddc04711d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:34.412578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412507 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412533 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412542 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412552 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgw76\" (UniqueName: \"kubernetes.io/projected/1eb937f4-88f8-40c9-bc40-615c9b81931a-kube-api-access-wgw76\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412578 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412562 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412590 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412599 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c432fb68-4436-42fe-86cc-eaddc04711d1-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412607 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c432fb68-4436-42fe-86cc-eaddc04711d1-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412616 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k866v\" (UniqueName: \"kubernetes.io/projected/c432fb68-4436-42fe-86cc-eaddc04711d1-kube-api-access-k866v\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412624 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb937f4-88f8-40c9-bc40-615c9b81931a-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412632 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c432fb68-4436-42fe-86cc-eaddc04711d1-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:34.412815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:34.412640 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb937f4-88f8-40c9-bc40-615c9b81931a-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:35.027611 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.027579 6319 patch_prober.go:29] interesting pod/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:22:35.027794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.027645 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:22:35.102892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.102813 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" Nov 03 16:22:35.103330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.102810 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9" event={"ID":"1eb937f4-88f8-40c9-bc40-615c9b81931a","Type":"ContainerDied","Data":"6ba6d2f62a01e634510f7d03ac154a25a620bdc835633e676aef76e51a73cef6"} Nov 03 16:22:35.103330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.102960 6319 scope.go:122] "RemoveContainer" containerID="30f912c888e4c9137fbd52e60908ed20603e0d678ec3955de3a7fc2a5d3e7527" Nov 03 16:22:35.104438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.104420 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" Nov 03 16:22:35.104542 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.104446 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p" event={"ID":"6115cfdb-b425-4cff-a4ee-82e5452f98f3","Type":"ContainerDied","Data":"a53f8c6c4049f940757e9bc73c7a5b2091de7dbed2b71098c558f5fd900ed0d5"} Nov 03 16:22:35.106092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.106068 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" event={"ID":"c432fb68-4436-42fe-86cc-eaddc04711d1","Type":"ContainerDied","Data":"b7deed67f7d7dfc94ea1fb63d4b973006f18728c6c54df25645a12908bc133cd"} Nov 03 16:22:35.106183 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.106075 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw" Nov 03 16:22:35.113832 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.113810 6319 scope.go:122] "RemoveContainer" containerID="30aad3b85fdcb959db4130db50a5dc869210c6dc097603738a7197e470d7252a" Nov 03 16:22:35.123642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.123628 6319 scope.go:122] "RemoveContainer" containerID="701bbbc36c3c37f4c094f9f9d5025a6c9c7d3457114f292d95154be14a09b798" Nov 03 16:22:35.133503 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.133483 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw"] Nov 03 16:22:35.139821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.139801 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-688sw"] Nov 03 16:22:35.151918 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.151896 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9"] Nov 03 16:22:35.156876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.156670 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-ffnc9"] Nov 03 16:22:35.168829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.168809 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p"] Nov 03 16:22:35.173662 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:35.173639 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-afb990-openshift-default-844489d9cf-dxk9p"] Nov 03 16:22:36.642961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:36.642932 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" path="/var/lib/kubelet/pods/1eb937f4-88f8-40c9-bc40-615c9b81931a/volumes" Nov 03 16:22:36.643480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:36.643462 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" path="/var/lib/kubelet/pods/6115cfdb-b425-4cff-a4ee-82e5452f98f3/volumes" Nov 03 16:22:36.643880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:36.643868 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" path="/var/lib/kubelet/pods/c432fb68-4436-42fe-86cc-eaddc04711d1/volumes" Nov 03 16:22:37.061914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.061862 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" containerID="cri-o://8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92" gracePeriod=45 Nov 03 16:22:37.119171 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.119135 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7"] Nov 03 16:22:37.119424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.119401 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" containerID="cri-o://ce90efc4493f3b43b3a3f1524d66e512840eb5bb7324b98e409a704b56924477" gracePeriod=30 Nov 03 16:22:37.126905 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.126883 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm"] Nov 03 16:22:37.127177 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.127157 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" containerID="cri-o://01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f" gracePeriod=30 Nov 03 16:22:37.136791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.136766 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx"] Nov 03 16:22:37.136950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.136932 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" containerID="cri-o://c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa" gracePeriod=30 Nov 03 16:22:37.145012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.144965 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9"] Nov 03 16:22:37.145187 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.145164 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" containerID="cri-o://c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860" gracePeriod=30 Nov 03 16:22:37.412943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.412914 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:22:37.557856 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.557829 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:22:37.562226 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562195 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir\") pod \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " Nov 03 16:22:37.562367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562257 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home\") pod \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " Nov 03 16:22:37.562367 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562294 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache\") pod \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " Nov 03 16:22:37.562502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562408 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir" pod "d278fa39-ab8f-4010-8e1f-d84e28f83e1f" (UID: "d278fa39-ab8f-4010-8e1f-d84e28f83e1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:37.562502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562420 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm\") pod \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " Nov 03 16:22:37.562594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562508 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home" pod "d278fa39-ab8f-4010-8e1f-d84e28f83e1f" (UID: "d278fa39-ab8f-4010-8e1f-d84e28f83e1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:37.562594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562578 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache" pod "d278fa39-ab8f-4010-8e1f-d84e28f83e1f" (UID: "d278fa39-ab8f-4010-8e1f-d84e28f83e1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:37.562594 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562577 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6\" (UniqueName: \"kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6\") pod \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " Nov 03 16:22:37.562741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.562649 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs\") pod \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\" (UID: \"d278fa39-ab8f-4010-8e1f-d84e28f83e1f\") " Nov 03 16:22:37.563218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.563198 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.563218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.563218 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.563396 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.563226 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.564970 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.564944 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs" pod "d278fa39-ab8f-4010-8e1f-d84e28f83e1f" (UID: "d278fa39-ab8f-4010-8e1f-d84e28f83e1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:37.565234 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.565216 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm" pod "d278fa39-ab8f-4010-8e1f-d84e28f83e1f" (UID: "d278fa39-ab8f-4010-8e1f-d84e28f83e1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:37.565296 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.565237 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6" pod "d278fa39-ab8f-4010-8e1f-d84e28f83e1f" (UID: "d278fa39-ab8f-4010-8e1f-d84e28f83e1f"). InnerVolumeSpecName "kube-api-access-bmfn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:37.663777 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.663691 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6\" (UniqueName: \"kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6\") pod \"cdd90afa-e149-404c-9c16-e3c34c463eec\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " Nov 03 16:22:37.663777 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.663740 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume\") pod \"cdd90afa-e149-404c-9c16-e3c34c463eec\" (UID: \"cdd90afa-e149-404c-9c16-e3c34c463eec\") " Nov 03 16:22:37.664303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.664021 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.664303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.664041 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmfn6\" (UniqueName: \"kubernetes.io/projected/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-kube-api-access-bmfn6\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.664303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.664054 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d278fa39-ab8f-4010-8e1f-d84e28f83e1f-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.664303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.664155 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume" pod "cdd90afa-e149-404c-9c16-e3c34c463eec" (UID: "cdd90afa-e149-404c-9c16-e3c34c463eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:37.666056 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.666032 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6" pod "cdd90afa-e149-404c-9c16-e3c34c463eec" (UID: "cdd90afa-e149-404c-9c16-e3c34c463eec"). InnerVolumeSpecName "kube-api-access-wmpm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:37.765348 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.765310 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpm6\" (UniqueName: \"kubernetes.io/projected/cdd90afa-e149-404c-9c16-e3c34c463eec-kube-api-access-wmpm6\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.765348 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.765341 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd90afa-e149-404c-9c16-e3c34c463eec-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:37.931175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:37.931076 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" containerID="cri-o://213b6916177c592d528c9bccdb3e97bd50d321f5d9579203e168fd58fde7646c" gracePeriod=45 Nov 03 16:22:38.136383 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.136348 6319 generic.go:356] "Generic (PLEG): container finished" podID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerID="7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4" exitCode=0 Nov 03 16:22:38.136554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.136406 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" Nov 03 16:22:38.136554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.136436 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" event={"ID":"cdd90afa-e149-404c-9c16-e3c34c463eec","Type":"ContainerDied","Data":"7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4"} Nov 03 16:22:38.136554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.136499 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v" event={"ID":"cdd90afa-e149-404c-9c16-e3c34c463eec","Type":"ContainerDied","Data":"57ba9fe91bca75e2e3224795388f4af47e22060f3b2662cfe0954b327c7c937f"} Nov 03 16:22:38.136554 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.136529 6319 scope.go:122] "RemoveContainer" containerID="7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4" Nov 03 16:22:38.138863 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.138835 6319 generic.go:356] "Generic (PLEG): container finished" podID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerID="213b6916177c592d528c9bccdb3e97bd50d321f5d9579203e168fd58fde7646c" exitCode=0 Nov 03 16:22:38.139012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.138935 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" event={"ID":"5759f2d4-c249-49ec-9b38-f38ebf64abfb","Type":"ContainerDied","Data":"213b6916177c592d528c9bccdb3e97bd50d321f5d9579203e168fd58fde7646c"} Nov 03 16:22:38.140574 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.140550 6319 generic.go:356] "Generic (PLEG): container finished" podID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerID="8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92" exitCode=0 Nov 03 16:22:38.140675 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.140622 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" event={"ID":"d278fa39-ab8f-4010-8e1f-d84e28f83e1f","Type":"ContainerDied","Data":"8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92"} Nov 03 16:22:38.140675 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.140649 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" event={"ID":"d278fa39-ab8f-4010-8e1f-d84e28f83e1f","Type":"ContainerDied","Data":"948a896d754ab165e8b25523fd868b6fe25a38ee5fab84f405567ba9bfafce77"} Nov 03 16:22:38.140675 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.140623 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8" Nov 03 16:22:38.149145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.149127 6319 scope.go:122] "RemoveContainer" containerID="7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4" Nov 03 16:22:38.149622 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:38.149594 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4\": container with ID starting with 7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4 not found: ID does not exist" containerID="7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4" Nov 03 16:22:38.149726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.149632 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4"} err="failed to get container status \"7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4\": rpc error: code = NotFound desc = could not find container \"7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4\": container with ID starting with 7304aa447876bc8b79f53ff25a10547736802565d36e2a18933638f0ed63c9b4 not found: ID does not exist" Nov 03 16:22:38.149726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.149717 6319 scope.go:122] "RemoveContainer" containerID="8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92" Nov 03 16:22:38.160188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.160172 6319 scope.go:122] "RemoveContainer" containerID="8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92" Nov 03 16:22:38.160494 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:38.160467 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92\": container with ID starting with 8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92 not found: ID does not exist" containerID="8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92" Nov 03 16:22:38.160577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.160504 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92"} err="failed to get container status \"8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92\": rpc error: code = NotFound desc = could not find container \"8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92\": container with ID starting with 8cd926960b06df5864b28db362b3f719fb97f99346b39233a7fe903818a58e92 not found: ID does not exist" Nov 03 16:22:38.173676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.173655 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v"] Nov 03 16:22:38.178117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.178095 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a19cd-98b5bd487-zm25v"] Nov 03 16:22:38.192807 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.192757 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:22:38.193378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.193361 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8"] Nov 03 16:22:38.198684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.198664 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-w2-b132e3-kserve-6f6f49b675-88zc8"] Nov 03 16:22:38.371517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371484 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm\") pod \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " Nov 03 16:22:38.371660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371525 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache\") pod \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " Nov 03 16:22:38.371660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371602 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs\") pod \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " Nov 03 16:22:38.371660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371621 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg\" (UniqueName: \"kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg\") pod \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " Nov 03 16:22:38.371751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371700 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir\") pod \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " Nov 03 16:22:38.371751 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371732 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home\") pod \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\" (UID: \"5759f2d4-c249-49ec-9b38-f38ebf64abfb\") " Nov 03 16:22:38.371858 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.371833 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache" pod "5759f2d4-c249-49ec-9b38-f38ebf64abfb" (UID: "5759f2d4-c249-49ec-9b38-f38ebf64abfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:38.372143 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.372113 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir" pod "5759f2d4-c249-49ec-9b38-f38ebf64abfb" (UID: "5759f2d4-c249-49ec-9b38-f38ebf64abfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:38.372143 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.372128 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home" pod "5759f2d4-c249-49ec-9b38-f38ebf64abfb" (UID: "5759f2d4-c249-49ec-9b38-f38ebf64abfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:38.372310 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.372169 6319 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-model-cache\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:38.373708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.373681 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm" pod "5759f2d4-c249-49ec-9b38-f38ebf64abfb" (UID: "5759f2d4-c249-49ec-9b38-f38ebf64abfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:38.373821 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.373804 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs" pod "5759f2d4-c249-49ec-9b38-f38ebf64abfb" (UID: "5759f2d4-c249-49ec-9b38-f38ebf64abfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:38.374069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.374040 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg" pod "5759f2d4-c249-49ec-9b38-f38ebf64abfb" (UID: "5759f2d4-c249-49ec-9b38-f38ebf64abfb"). InnerVolumeSpecName "kube-api-access-qn8qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:38.473791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.473757 6319 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-dshm\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:38.473791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.473787 6319 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tls-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:38.473967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.473802 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8qg\" (UniqueName: \"kubernetes.io/projected/5759f2d4-c249-49ec-9b38-f38ebf64abfb-kube-api-access-qn8qg\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:38.473967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.473817 6319 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-tmp-dir\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:38.473967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.473829 6319 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5759f2d4-c249-49ec-9b38-f38ebf64abfb-home\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:38.642545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.642515 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" path="/var/lib/kubelet/pods/cdd90afa-e149-404c-9c16-e3c34c463eec/volumes" Nov 03 16:22:38.642950 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:38.642935 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" path="/var/lib/kubelet/pods/d278fa39-ab8f-4010-8e1f-d84e28f83e1f/volumes" Nov 03 16:22:39.148642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.148613 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" Nov 03 16:22:39.148642 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.148622 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr" event={"ID":"5759f2d4-c249-49ec-9b38-f38ebf64abfb","Type":"ContainerDied","Data":"0a0be9f93d235773c9a1ca6fde8e1fab98e776f328b75795552d8699e1f0f00d"} Nov 03 16:22:39.149172 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.148671 6319 scope.go:122] "RemoveContainer" containerID="213b6916177c592d528c9bccdb3e97bd50d321f5d9579203e168fd58fde7646c" Nov 03 16:22:39.169198 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.169174 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr"] Nov 03 16:22:39.173387 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.173363 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b132e3/distinct-2-w2-b132e3-kserve-c9566fc48-2dkdr"] Nov 03 16:22:39.928165 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.928127 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct"] Nov 03 16:22:39.928438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:39.928409 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" podUID="7e9234b6-d71b-48ab-b92f-29cb98ab9352" containerName="maas-api" containerID="cri-o://f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d" gracePeriod=30 Nov 03 16:22:40.473949 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.473886 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:22:40.497315 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.497275 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs\") pod \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " Nov 03 16:22:40.497497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.497350 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls\") pod \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " Nov 03 16:22:40.497497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.497450 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch\" (UniqueName: \"kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch\") pod \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " Nov 03 16:22:40.497702 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.497528 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca\") pod \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\" (UID: \"7e9234b6-d71b-48ab-b92f-29cb98ab9352\") " Nov 03 16:22:40.498015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.497969 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca" pod "7e9234b6-d71b-48ab-b92f-29cb98ab9352" (UID: "7e9234b6-d71b-48ab-b92f-29cb98ab9352"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:40.499808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.499660 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9234b6-d71b-48ab-b92f-29cb98ab9352-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:40.500342 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.500307 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs" pod "7e9234b6-d71b-48ab-b92f-29cb98ab9352" (UID: "7e9234b6-d71b-48ab-b92f-29cb98ab9352"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:40.501620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.501596 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch" pod "7e9234b6-d71b-48ab-b92f-29cb98ab9352" (UID: "7e9234b6-d71b-48ab-b92f-29cb98ab9352"). InnerVolumeSpecName "kube-api-access-fzpch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:40.502557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.502524 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls" pod "7e9234b6-d71b-48ab-b92f-29cb98ab9352" (UID: "7e9234b6-d71b-48ab-b92f-29cb98ab9352"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:40.601251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.601217 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:40.601251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.601247 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9234b6-d71b-48ab-b92f-29cb98ab9352-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:40.601448 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.601262 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzpch\" (UniqueName: \"kubernetes.io/projected/7e9234b6-d71b-48ab-b92f-29cb98ab9352-kube-api-access-fzpch\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:40.642417 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:40.642386 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" path="/var/lib/kubelet/pods/5759f2d4-c249-49ec-9b38-f38ebf64abfb/volumes" Nov 03 16:22:41.170617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.170579 6319 generic.go:356] "Generic (PLEG): container finished" podID="7e9234b6-d71b-48ab-b92f-29cb98ab9352" containerID="f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d" exitCode=0 Nov 03 16:22:41.170791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.170633 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" Nov 03 16:22:41.170791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.170642 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" event={"ID":"7e9234b6-d71b-48ab-b92f-29cb98ab9352","Type":"ContainerDied","Data":"f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d"} Nov 03 16:22:41.170791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.170692 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct" event={"ID":"7e9234b6-d71b-48ab-b92f-29cb98ab9352","Type":"ContainerDied","Data":"1f3672a98281322a43760045f3a33d152d09cc59faa4421848b1a1f5762ec35c"} Nov 03 16:22:41.170791 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.170722 6319 scope.go:122] "RemoveContainer" containerID="f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d" Nov 03 16:22:41.180653 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.180635 6319 scope.go:122] "RemoveContainer" containerID="f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d" Nov 03 16:22:41.180951 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:41.180932 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d\": container with ID starting with f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d not found: ID does not exist" containerID="f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d" Nov 03 16:22:41.181042 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.180963 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d"} err="failed to get container status \"f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d\": rpc error: code = NotFound desc = could not find container \"f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d\": container with ID starting with f9a323f0b2bf70349f9a5b1102ab0c2e62c771bf8f787668b5efaa491482e92d not found: ID does not exist" Nov 03 16:22:41.232136 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.232088 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct"] Nov 03 16:22:41.237269 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:41.237243 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b132e3-664c7d48dc-4bwct"] Nov 03 16:22:42.231333 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.231299 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" event={"ID":"ca99b6fc-2758-4f70-bd99-3728b13606b6","Type":"ContainerDied","Data":"ce90efc4493f3b43b3a3f1524d66e512840eb5bb7324b98e409a704b56924477"} Nov 03 16:22:42.232066 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.232039 6319 generic.go:356] "Generic (PLEG): container finished" podID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerID="ce90efc4493f3b43b3a3f1524d66e512840eb5bb7324b98e409a704b56924477" exitCode=0 Nov 03 16:22:42.468954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.468297 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:22:42.502306 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.502221 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:22:42.533493 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.533464 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:22:42.560052 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560022 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.560217 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560081 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.560217 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560165 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.560323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560254 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.560323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560270 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.560323 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560317 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.560472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560351 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.560472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560385 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.560472 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560427 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.560617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560493 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.560617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560503 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:42.560617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560551 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.560617 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560587 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.560804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560640 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.560804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560647 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.560804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560698 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.560804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560753 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.560804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560783 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.561047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560815 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.561047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560847 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.561047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560877 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.561047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.560914 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.561047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561009 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.561047 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561037 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.561517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561073 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.561517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561124 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs\") pod \"7cac05b9-289b-4776-a5b1-768140d2d0a8\" (UID: \"7cac05b9-289b-4776-a5b1-768140d2d0a8\") " Nov 03 16:22:42.561517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561164 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.561517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561204 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.561517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561254 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token\") pod \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\" (UID: \"1a3383e3-d49a-4ea2-9f08-2e466fad2c47\") " Nov 03 16:22:42.561517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561286 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket\") pod \"564b07ef-7d46-4fe2-b764-3edd43fba364\" (UID: \"564b07ef-7d46-4fe2-b764-3edd43fba364\") " Nov 03 16:22:42.561808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561750 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.561808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561773 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cac05b9-289b-4776-a5b1-768140d2d0a8-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.561808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561787 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.562331 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561008 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562331 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561182 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561205 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561480 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561473 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561665 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561823 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.561946 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.562157 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.562398 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.562506 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.562299 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.563080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.562520 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.563421 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.563380 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:42.564115 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.564038 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "kube-api-access-v9q4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.564357 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.564333 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "kube-api-access-rrmqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.566087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.566062 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.566087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.566074 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:42.566338 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.566253 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.566432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.566408 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "kube-api-access-wb2xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.566518 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.566456 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy" pod "1a3383e3-d49a-4ea2-9f08-2e466fad2c47" (UID: "1a3383e3-d49a-4ea2-9f08-2e466fad2c47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.566749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.566730 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.567068 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.567042 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token" pod "7cac05b9-289b-4776-a5b1-768140d2d0a8" (UID: "7cac05b9-289b-4776-a5b1-768140d2d0a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.567429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.567407 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.567946 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.567925 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo" pod "564b07ef-7d46-4fe2-b764-3edd43fba364" (UID: "564b07ef-7d46-4fe2-b764-3edd43fba364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:42.643568 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.643539 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9234b6-d71b-48ab-b92f-29cb98ab9352" path="/var/lib/kubelet/pods/7e9234b6-d71b-48ab-b92f-29cb98ab9352/volumes" Nov 03 16:22:42.662646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662623 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmqw\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-kube-api-access-rrmqw\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662648 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662659 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9q4x\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-kube-api-access-v9q4x\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662668 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662676 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662686 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662694 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662703 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662710 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662719 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662726 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662734 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662742 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662750 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564b07ef-7d46-4fe2-b764-3edd43fba364-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662759 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.662765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662767 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662775 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662783 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662792 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cac05b9-289b-4776-a5b1-768140d2d0a8-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662802 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662809 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662829 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3383e3-d49a-4ea2-9f08-2e466fad2c47-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662838 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564b07ef-7d46-4fe2-b764-3edd43fba364-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.663173 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.662847 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb2xw\" (UniqueName: \"kubernetes.io/projected/7cac05b9-289b-4776-a5b1-768140d2d0a8-kube-api-access-wb2xw\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.773452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.773425 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:22:42.864737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.864708 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.864890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.864752 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.864890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.864779 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.864890 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.864810 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.865081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.864935 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.865081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865022 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.865081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865034 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.865346 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865150 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.865346 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865199 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.865346 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865212 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:42.865346 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865256 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket\") pod \"ca99b6fc-2758-4f70-bd99-3728b13606b6\" (UID: \"ca99b6fc-2758-4f70-bd99-3728b13606b6\") " Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865471 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865495 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865596 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865798 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865840 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865856 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865870 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.866351 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.865904 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca99b6fc-2758-4f70-bd99-3728b13606b6-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.867637 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.867611 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:42.867734 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.867641 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:42.867807 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.867757 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "kube-api-access-pkcwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.868676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.868655 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token" pod "ca99b6fc-2758-4f70-bd99-3728b13606b6" (UID: "ca99b6fc-2758-4f70-bd99-3728b13606b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:42.966590 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.966555 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkcwb\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-kube-api-access-pkcwb\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.966590 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.966590 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.966783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.966601 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:42.966783 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:42.966610 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca99b6fc-2758-4f70-bd99-3728b13606b6-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:43.239978 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.239885 6319 generic.go:356] "Generic (PLEG): container finished" podID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerID="c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa" exitCode=0 Nov 03 16:22:43.239978 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.239943 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" Nov 03 16:22:43.240535 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.239972 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" event={"ID":"7cac05b9-289b-4776-a5b1-768140d2d0a8","Type":"ContainerDied","Data":"c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa"} Nov 03 16:22:43.240535 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.240041 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx" event={"ID":"7cac05b9-289b-4776-a5b1-768140d2d0a8","Type":"ContainerDied","Data":"0f89b3698d280e884a842247674fffb4b8a51325e4a27e22f7edeb373ffaf008"} Nov 03 16:22:43.240535 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.240065 6319 scope.go:122] "RemoveContainer" containerID="c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa" Nov 03 16:22:43.242498 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.242342 6319 generic.go:356] "Generic (PLEG): container finished" podID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerID="01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f" exitCode=0 Nov 03 16:22:43.242498 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.242401 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" Nov 03 16:22:43.242666 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.242581 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" event={"ID":"1a3383e3-d49a-4ea2-9f08-2e466fad2c47","Type":"ContainerDied","Data":"01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f"} Nov 03 16:22:43.242666 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.242650 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm" event={"ID":"1a3383e3-d49a-4ea2-9f08-2e466fad2c47","Type":"ContainerDied","Data":"cdcf23ea487f0b3bbfe1cd81135f77083e437c755d899a9dd7552c3a5651e895"} Nov 03 16:22:43.244079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.244045 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" Nov 03 16:22:43.244204 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.244126 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7" event={"ID":"ca99b6fc-2758-4f70-bd99-3728b13606b6","Type":"ContainerDied","Data":"259a37a1655f72f0caba7aa04be962e29a3e3b9fa9b8f2546bdafccc5b6e5d2c"} Nov 03 16:22:43.245674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.245649 6319 generic.go:356] "Generic (PLEG): container finished" podID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerID="c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860" exitCode=0 Nov 03 16:22:43.245765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.245684 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" Nov 03 16:22:43.245765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.245717 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" event={"ID":"564b07ef-7d46-4fe2-b764-3edd43fba364","Type":"ContainerDied","Data":"c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860"} Nov 03 16:22:43.245765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.245756 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9" event={"ID":"564b07ef-7d46-4fe2-b764-3edd43fba364","Type":"ContainerDied","Data":"cfcba4d6a4ada311a6c8de2619b978f54ba66c5612ab8074b3774c4c58d3c1d0"} Nov 03 16:22:43.251291 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.251131 6319 scope.go:122] "RemoveContainer" containerID="c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa" Nov 03 16:22:43.251434 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:43.251417 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa\": container with ID starting with c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa not found: ID does not exist" containerID="c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa" Nov 03 16:22:43.251476 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.251440 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa"} err="failed to get container status \"c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa\": rpc error: code = NotFound desc = could not find container \"c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa\": container with ID starting with c51e565c740306cdd281305703c6828640235881ba884c320036741b52a2adfa not found: ID does not exist" Nov 03 16:22:43.251476 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.251459 6319 scope.go:122] "RemoveContainer" containerID="01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f" Nov 03 16:22:43.260749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.260731 6319 scope.go:122] "RemoveContainer" containerID="d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f" Nov 03 16:22:43.269561 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.269539 6319 scope.go:122] "RemoveContainer" containerID="01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f" Nov 03 16:22:43.269830 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:43.269804 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f\": container with ID starting with 01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f not found: ID does not exist" containerID="01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f" Nov 03 16:22:43.269889 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.269841 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f"} err="failed to get container status \"01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f\": rpc error: code = NotFound desc = could not find container \"01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f\": container with ID starting with 01107d81f2145a1078a433e4f27fb3571bab02146a5693a41a9194f3f77bdc8f not found: ID does not exist" Nov 03 16:22:43.269889 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.269863 6319 scope.go:122] "RemoveContainer" containerID="d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f" Nov 03 16:22:43.270150 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:43.270130 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f\": container with ID starting with d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f not found: ID does not exist" containerID="d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f" Nov 03 16:22:43.270234 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.270158 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f"} err="failed to get container status \"d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f\": rpc error: code = NotFound desc = could not find container \"d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f\": container with ID starting with d03fa6973c9fc311631014f92a0a795ca4bbdeb4263d93a3ec02a4f8d8b6710f not found: ID does not exist" Nov 03 16:22:43.270234 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.270179 6319 scope.go:122] "RemoveContainer" containerID="ce90efc4493f3b43b3a3f1524d66e512840eb5bb7324b98e409a704b56924477" Nov 03 16:22:43.271563 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.271547 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm"] Nov 03 16:22:43.276577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.276558 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-rrmxm"] Nov 03 16:22:43.279482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.279467 6319 scope.go:122] "RemoveContainer" containerID="c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860" Nov 03 16:22:43.291520 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.291496 6319 scope.go:122] "RemoveContainer" containerID="c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860" Nov 03 16:22:43.292131 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:43.292100 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860\": container with ID starting with c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860 not found: ID does not exist" containerID="c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860" Nov 03 16:22:43.292215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.292139 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860"} err="failed to get container status \"c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860\": rpc error: code = NotFound desc = could not find container \"c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860\": container with ID starting with c786fab80cf42e4449f433ab862775b3474863a4c67db6c3d85145c5df72a860 not found: ID does not exist" Nov 03 16:22:43.294232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.294171 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx"] Nov 03 16:22:43.296178 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.296161 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-95khx"] Nov 03 16:22:43.326257 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.326223 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7"] Nov 03 16:22:43.331559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.331535 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-7g9n7"] Nov 03 16:22:43.344592 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.344569 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9"] Nov 03 16:22:43.351653 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:43.351630 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a19cd-openshift-default-785b9fcf97-swkl9"] Nov 03 16:22:44.641999 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:44.641950 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" path="/var/lib/kubelet/pods/1a3383e3-d49a-4ea2-9f08-2e466fad2c47/volumes" Nov 03 16:22:44.642542 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:44.642523 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" path="/var/lib/kubelet/pods/564b07ef-7d46-4fe2-b764-3edd43fba364/volumes" Nov 03 16:22:44.642966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:44.642953 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" path="/var/lib/kubelet/pods/7cac05b9-289b-4776-a5b1-768140d2d0a8/volumes" Nov 03 16:22:44.643688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:44.643673 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" path="/var/lib/kubelet/pods/ca99b6fc-2758-4f70-bd99-3728b13606b6/volumes" Nov 03 16:22:47.294792 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.294758 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js"] Nov 03 16:22:47.295357 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.294971 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" containerID="cri-o://377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903" gracePeriod=30 Nov 03 16:22:47.301726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.301702 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x"] Nov 03 16:22:47.301914 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.301895 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" containerID="cri-o://eb081d5e3b1dd92edf25f114e4a21ca7f4c90f6be3ce12491c3bee3b49231965" gracePeriod=30 Nov 03 16:22:47.314475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.314453 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9"] Nov 03 16:22:47.314649 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.314629 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" containerID="cri-o://8d36690bee9fae901faa4e7e2dec9319f6dfdd15aaf9d5bbb58d27463cf62275" gracePeriod=30 Nov 03 16:22:47.324896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.324873 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh"] Nov 03 16:22:47.325089 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.325070 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" containerID="cri-o://9fdc3327c8ffddb7bc47dcda2167b8ba5978350118fdaa4b8061fd65b05963a1" gracePeriod=30 Nov 03 16:22:47.436734 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.436697 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:47.436925 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.436746 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:47.689799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.689721 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:47.689799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:47.689770 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:48.603270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:48.603236 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:48.603664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:48.603303 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:48.776722 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:48.776688 6319 patch_prober.go:29] interesting pod/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:22:48.776876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:48.776756 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:22:50.092700 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.092666 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4b666674-mnhsn"] Nov 03 16:22:50.093177 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.092872 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-c4b666674-mnhsn" podUID="d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" containerName="manager" containerID="cri-o://cb46079233f7679e18b78c1126573d05dcf765af8cef581e3a76df3e50c8507c" gracePeriod=30 Nov 03 16:22:50.299246 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.299215 6319 generic.go:356] "Generic (PLEG): container finished" podID="d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" containerID="cb46079233f7679e18b78c1126573d05dcf765af8cef581e3a76df3e50c8507c" exitCode=0 Nov 03 16:22:50.299411 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.299288 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-mnhsn" event={"ID":"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0","Type":"ContainerDied","Data":"cb46079233f7679e18b78c1126573d05dcf765af8cef581e3a76df3e50c8507c"} Nov 03 16:22:50.342623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.342598 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:22:50.444959 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.444886 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs\") pod \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " Nov 03 16:22:50.445117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.445004 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv\" (UniqueName: \"kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv\") pod \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " Nov 03 16:22:50.445117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.445071 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert\") pod \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\" (UID: \"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0\") " Nov 03 16:22:50.447037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.446971 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs" pod "d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" (UID: "d12a99c4-51ee-4f05-8cf4-e8cee1981bc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:50.447205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.447181 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert" pod "d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" (UID: "d12a99c4-51ee-4f05-8cf4-e8cee1981bc0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:22:50.447270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.447259 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv" pod "d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" (UID: "d12a99c4-51ee-4f05-8cf4-e8cee1981bc0"). InnerVolumeSpecName "kube-api-access-xbmvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:50.546803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.546775 6319 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-webhook-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:50.546803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.546800 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:50.546803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:50.546809 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbmvv\" (UniqueName: \"kubernetes.io/projected/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0-kube-api-access-xbmvv\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:51.308329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:51.308292 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-mnhsn" Nov 03 16:22:51.308329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:51.308301 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-mnhsn" event={"ID":"d12a99c4-51ee-4f05-8cf4-e8cee1981bc0","Type":"ContainerDied","Data":"e989a97070de290d77abcac7a1f46d945bdd2c38291b202fe85c0099f2c8fa7d"} Nov 03 16:22:51.308891 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:51.308371 6319 scope.go:122] "RemoveContainer" containerID="cb46079233f7679e18b78c1126573d05dcf765af8cef581e3a76df3e50c8507c" Nov 03 16:22:51.332313 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:51.332284 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4b666674-mnhsn"] Nov 03 16:22:51.334136 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:51.334110 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4b666674-mnhsn"] Nov 03 16:22:52.030840 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.029514 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4b666674-sbcxm"] Nov 03 16:22:52.033555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.033512 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" Nov 03 16:22:52.033793 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.033755 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" Nov 03 16:22:52.033929 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.033918 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" Nov 03 16:22:52.034110 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034096 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" Nov 03 16:22:52.034208 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034199 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" Nov 03 16:22:52.034287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034278 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" Nov 03 16:22:52.034363 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034355 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" Nov 03 16:22:52.034435 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034426 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" Nov 03 16:22:52.034518 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034509 6319 cpu_manager.go:414] "containerMap: removing container" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" Nov 03 16:22:52.034584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034574 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" Nov 03 16:22:52.034656 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034648 6319 cpu_manager.go:414] "containerMap: removing container" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" Nov 03 16:22:52.034727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034717 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" Nov 03 16:22:52.034804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034794 6319 cpu_manager.go:414] "containerMap: removing container" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" Nov 03 16:22:52.034876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034867 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" Nov 03 16:22:52.034967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.034958 6319 cpu_manager.go:414] "containerMap: removing container" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" Nov 03 16:22:52.035076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035064 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035147 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035157 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035167 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035174 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035184 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035191 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035202 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035209 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035222 6319 cpu_manager.go:414] "containerMap: removing container" podUID="7e9234b6-d71b-48ab-b92f-29cb98ab9352" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035228 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9234b6-d71b-48ab-b92f-29cb98ab9352" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035241 6319 cpu_manager.go:414] "containerMap: removing container" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035247 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035257 6319 cpu_manager.go:414] "containerMap: removing container" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035264 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035285 6319 cpu_manager.go:414] "containerMap: removing container" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035292 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035300 6319 cpu_manager.go:414] "containerMap: removing container" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035307 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035319 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035326 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035336 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035343 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035354 6319 cpu_manager.go:414] "containerMap: removing container" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035398 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035411 6319 cpu_manager.go:414] "containerMap: removing container" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035419 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035429 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035436 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035446 6319 cpu_manager.go:414] "containerMap: removing container" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035453 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035461 6319 cpu_manager.go:414] "containerMap: removing container" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035468 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035478 6319 cpu_manager.go:414] "containerMap: removing container" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035485 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035514 6319 cpu_manager.go:414] "containerMap: removing container" podUID="34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" containerName="revoke-keys" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035521 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" containerName="revoke-keys" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035530 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035536 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035547 6319 cpu_manager.go:414] "containerMap: removing container" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035556 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035585 6319 cpu_manager.go:414] "containerMap: removing container" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035593 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035609 6319 cpu_manager.go:414] "containerMap: removing container" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035617 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035629 6319 cpu_manager.go:414] "containerMap: removing container" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerName="payload-pre-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035636 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerName="payload-pre-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035663 6319 cpu_manager.go:414] "containerMap: removing container" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035671 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035683 6319 cpu_manager.go:414] "containerMap: removing container" podUID="5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035690 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035701 6319 cpu_manager.go:414] "containerMap: removing container" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035707 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035715 6319 cpu_manager.go:414] "containerMap: removing container" podUID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035722 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035733 6319 cpu_manager.go:414] "containerMap: removing container" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035740 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035751 6319 cpu_manager.go:414] "containerMap: removing container" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035758 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035766 6319 cpu_manager.go:414] "containerMap: removing container" podUID="d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" containerName="manager" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035773 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" containerName="manager" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035789 6319 cpu_manager.go:414] "containerMap: removing container" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035796 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035805 6319 cpu_manager.go:414] "containerMap: removing container" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035812 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035942 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f71d83-d648-4a63-9289-60904622b040" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035958 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b4cfda1-4aaa-4c6d-ad0b-96b0d05f7bd4" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035969 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="d278fa39-ab8f-4010-8e1f-d84e28f83e1f" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.035999 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="def1b0ba-39a7-47ed-9dcd-bc84b2d14410" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036008 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="5759f2d4-c249-49ec-9b38-f38ebf64abfb" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036017 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca99b6fc-2758-4f70-bd99-3728b13606b6" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036027 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" containerName="manager" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036037 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ecd076-c57d-4ba5-874c-70c829037b02" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036046 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="564b07ef-7d46-4fe2-b764-3edd43fba364" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036054 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c432fb68-4436-42fe-86cc-eaddc04711d1" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036066 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54ad8dd-f076-4c92-abe2-6366a3e7d803" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036076 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9214762-f1a2-4155-b8a4-6b01cadc36da" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036085 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036093 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="46270e98-843d-4adc-9950-4bcf08c9f436" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036101 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b81517c-8ea1-47ba-ac5d-f45983c8df96" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036118 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="effc8f7a-b111-43ba-ae49-cc7698dfe3c2" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036132 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036139 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036149 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036157 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3383e3-d49a-4ea2-9f08-2e466fad2c47" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036165 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19cf4b4-48e2-4276-9a4a-f76af7843f29" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036174 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9234b6-d71b-48ab-b92f-29cb98ab9352" containerName="maas-api" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036182 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d44abb0-b332-4714-9585-0be6d576d530" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036191 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b42ef8-935c-4a0d-b01e-848f3bc493bb" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036200 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a0bf2d-0d7c-457c-9be4-b6e08db98c91" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036208 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cac05b9-289b-4776-a5b1-768140d2d0a8" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036216 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb937f4-88f8-40c9-bc40-615c9b81931a" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036225 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8316fc-748b-4d59-9075-7bb4be758b7c" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036237 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa1a461-d6d4-49dd-8c8c-c1ab8d647df5" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036246 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2451b7c-c27c-40b7-8e1f-ad322468b1d9" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036253 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5e6618-836c-4f45-8ef3-7e9aaa552773" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036262 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3595af-163a-4339-858f-4f1885ce04a2" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036270 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="988c231f-9d8e-4873-8e01-8599e5d904d6" containerName="payload-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036277 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" containerName="revoke-keys" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036286 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1605541c-2e9c-42fd-9cd0-1606e0606e83" containerName="main" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036295 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6251865-47a9-408e-a9eb-9546ec0e4175" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036304 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="6115cfdb-b425-4cff-a4ee-82e5452f98f3" containerName="istio-proxy" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036313 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c01d5a-fc60-4aee-89be-50ecd3abb448" containerName="payload-pre-processing" Nov 03 16:22:52.039865 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.036323 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd90afa-e149-404c-9c16-e3c34c463eec" containerName="payload-pre-processing" Nov 03 16:22:52.064841 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.064757 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4b666674-sbcxm"] Nov 03 16:22:52.065306 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.065230 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.072371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.072079 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 16:22:52.072371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.072153 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 16:22:52.072371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.072241 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7z7w6\"" Nov 03 16:22:52.072371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.072341 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 16:22:52.165669 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.165618 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.166215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.166156 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xww\" (UniqueName: \"kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.166452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.166421 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.267680 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.267642 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xww\" (UniqueName: \"kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.268111 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.268081 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.269347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.269328 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.281620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.281540 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.282466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.282439 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.283602 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.283579 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xww\" (UniqueName: \"kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww\") pod \"maas-controller-c4b666674-sbcxm\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.406597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.406330 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:52.703776 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:52.703372 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12a99c4-51ee-4f05-8cf4-e8cee1981bc0" path="/var/lib/kubelet/pods/d12a99c4-51ee-4f05-8cf4-e8cee1981bc0/volumes" Nov 03 16:22:53.110069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.109722 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284635 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284691 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284741 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284772 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284810 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284845 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.284916 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.285014 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.288368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.285137 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs\") pod \"d380c991-e43f-48a7-aa95-d4bd93207880\" (UID: \"d380c991-e43f-48a7-aa95-d4bd93207880\") " Nov 03 16:22:53.301626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.300248 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.301626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.300599 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.301626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.301526 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.301626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.301591 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.303294 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.303252 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:53.306861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.306820 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.310394 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.310372 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:53.330555 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.330480 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.351009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.342422 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6" pod "d380c991-e43f-48a7-aa95-d4bd93207880" (UID: "d380c991-e43f-48a7-aa95-d4bd93207880"). InnerVolumeSpecName "kube-api-access-927h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399740 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399781 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399797 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399814 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d380c991-e43f-48a7-aa95-d4bd93207880-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399829 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399844 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d380c991-e43f-48a7-aa95-d4bd93207880-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399859 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d380c991-e43f-48a7-aa95-d4bd93207880-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399873 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.399927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.399891 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927h6\" (UniqueName: \"kubernetes.io/projected/d380c991-e43f-48a7-aa95-d4bd93207880-kube-api-access-927h6\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.411152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.409546 6319 generic.go:356] "Generic (PLEG): container finished" podID="d380c991-e43f-48a7-aa95-d4bd93207880" containerID="377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903" exitCode=0 Nov 03 16:22:53.411152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.409723 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" Nov 03 16:22:53.411152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.409925 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" event={"ID":"d380c991-e43f-48a7-aa95-d4bd93207880","Type":"ContainerDied","Data":"377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903"} Nov 03 16:22:53.411152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.409971 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js" event={"ID":"d380c991-e43f-48a7-aa95-d4bd93207880","Type":"ContainerDied","Data":"12560a731964ce0c2c85c993a9be6c936c470b7370ba33bcf048ab1baa4a23b5"} Nov 03 16:22:53.411152 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.410018 6319 scope.go:122] "RemoveContainer" containerID="377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903" Nov 03 16:22:53.559106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.505618 6319 generic.go:356] "Generic (PLEG): container finished" podID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerID="eb081d5e3b1dd92edf25f114e4a21ca7f4c90f6be3ce12491c3bee3b49231965" exitCode=0 Nov 03 16:22:53.559106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.505739 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" event={"ID":"94cc12de-8730-4db9-a515-5ecd5d85d28f","Type":"ContainerDied","Data":"eb081d5e3b1dd92edf25f114e4a21ca7f4c90f6be3ce12491c3bee3b49231965"} Nov 03 16:22:53.611015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.610912 6319 scope.go:122] "RemoveContainer" containerID="377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903" Nov 03 16:22:53.637019 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:53.611846 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903\": container with ID starting with 377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903 not found: ID does not exist" containerID="377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903" Nov 03 16:22:53.637019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.611887 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903"} err="failed to get container status \"377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903\": rpc error: code = NotFound desc = could not find container \"377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903\": container with ID starting with 377e0bf10bb3b042303f59be0f768d3ea275514a2b4275d5e71c397c56b07903 not found: ID does not exist" Nov 03 16:22:53.681689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.668956 6319 generic.go:356] "Generic (PLEG): container finished" podID="1375f130-dfc4-4340-b574-b539dca36dc5" containerID="8d36690bee9fae901faa4e7e2dec9319f6dfdd15aaf9d5bbb58d27463cf62275" exitCode=0 Nov 03 16:22:53.681689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.669167 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" event={"ID":"1375f130-dfc4-4340-b574-b539dca36dc5","Type":"ContainerDied","Data":"8d36690bee9fae901faa4e7e2dec9319f6dfdd15aaf9d5bbb58d27463cf62275"} Nov 03 16:22:53.681689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.673758 6319 generic.go:356] "Generic (PLEG): container finished" podID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerID="9fdc3327c8ffddb7bc47dcda2167b8ba5978350118fdaa4b8061fd65b05963a1" exitCode=0 Nov 03 16:22:53.681689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.673834 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" event={"ID":"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76","Type":"ContainerDied","Data":"9fdc3327c8ffddb7bc47dcda2167b8ba5978350118fdaa4b8061fd65b05963a1"} Nov 03 16:22:53.681689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.677871 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:22:53.681689 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.680250 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:22:53.695920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.683909 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4b666674-sbcxm"] Nov 03 16:22:53.704452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.704381 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:22:53.777309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.777271 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.777834 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.777730 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.778100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.778055 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.778375 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.778324 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.778613 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.778506 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.778613 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.778586 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.779452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.779047 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.779452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.779393 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.780931 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.780527 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:53.781652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.781548 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.781652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.781613 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.781964 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.781906 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.782246 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.781943 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.782403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.782362 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.782560 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.782524 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.782738 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.782682 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.782876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.782837 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.783097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.782974 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.783097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783032 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.783097 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783071 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.783482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783337 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.783482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783373 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.783482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783411 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.783482 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783441 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.784125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783763 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs\") pod \"94cc12de-8730-4db9-a515-5ecd5d85d28f\" (UID: \"94cc12de-8730-4db9-a515-5ecd5d85d28f\") " Nov 03 16:22:53.784125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783801 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.784125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783837 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.784125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783870 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.784125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783933 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo\") pod \"1375f130-dfc4-4340-b574-b539dca36dc5\" (UID: \"1375f130-dfc4-4340-b574-b539dca36dc5\") " Nov 03 16:22:53.784125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.783968 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs\") pod \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\" (UID: \"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76\") " Nov 03 16:22:53.785241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.785036 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.785241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.785065 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.785241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.785081 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375f130-dfc4-4340-b574-b539dca36dc5-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.785529 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.785469 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.786100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.785790 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.788083 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.787828 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:53.788653 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.788495 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.788951 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.788788 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.789355 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.789101 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.789678 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.789484 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.790064 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.789802 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.795958 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.795654 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "kube-api-access-qcfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.798622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.798253 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.800644 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.800615 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.800747 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.800690 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.801157 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.801129 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.801258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.801195 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.801258 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.801229 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:22:53.801379 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.801295 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.801585 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.801549 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:53.803684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.803654 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js"] Nov 03 16:22:53.804148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.804123 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:53.804618 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.804578 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.804851 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.804829 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-wh4js"] Nov 03 16:22:53.805070 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.805037 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.805432 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.805400 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo" pod "1375f130-dfc4-4340-b574-b539dca36dc5" (UID: "1375f130-dfc4-4340-b574-b539dca36dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:22:53.805635 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.805605 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4" pod "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" (UID: "cb899c93-ba2a-4a6f-98ef-5c8cdabefa76"). InnerVolumeSpecName "kube-api-access-d45q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.805848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.805829 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:22:53.806241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.806216 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7" pod "94cc12de-8730-4db9-a515-5ecd5d85d28f" (UID: "94cc12de-8730-4db9-a515-5ecd5d85d28f"). InnerVolumeSpecName "kube-api-access-zjxm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886592 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886621 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886630 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886639 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886647 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886656 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886664 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d45q4\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-kube-api-access-d45q4\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886674 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxm7\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-kube-api-access-zjxm7\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886682 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886690 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc12de-8730-4db9-a515-5ecd5d85d28f-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.886692 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886698 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886706 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886714 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886725 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886733 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc12de-8730-4db9-a515-5ecd5d85d28f-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886744 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886758 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886767 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc12de-8730-4db9-a515-5ecd5d85d28f-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886776 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886784 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375f130-dfc4-4340-b574-b539dca36dc5-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886792 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886802 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375f130-dfc4-4340-b574-b539dca36dc5-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886811 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:53.887474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:53.886819 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/1375f130-dfc4-4340-b574-b539dca36dc5-kube-api-access-qcfmg\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:22:54.641864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.641828 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" path="/var/lib/kubelet/pods/d380c991-e43f-48a7-aa95-d4bd93207880/volumes" Nov 03 16:22:54.681228 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.681199 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-sbcxm" event={"ID":"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a","Type":"ContainerStarted","Data":"74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c"} Nov 03 16:22:54.681344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.681246 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-sbcxm" event={"ID":"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a","Type":"ContainerStarted","Data":"83a661dde19a9956cb45708ebf74a4f9f01e22694475d9024149d2327098e65c"} Nov 03 16:22:54.681411 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.681370 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:22:54.683676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.683653 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" Nov 03 16:22:54.683779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.683669 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x" event={"ID":"94cc12de-8730-4db9-a515-5ecd5d85d28f","Type":"ContainerDied","Data":"17ee0098678e08d41ae6e9120a0f6ff6d38736906ca897627fbc5effca6c5d0d"} Nov 03 16:22:54.683779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.683740 6319 scope.go:122] "RemoveContainer" containerID="eb081d5e3b1dd92edf25f114e4a21ca7f4c90f6be3ce12491c3bee3b49231965" Nov 03 16:22:54.685148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.685078 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" Nov 03 16:22:54.685148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.685098 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9" event={"ID":"1375f130-dfc4-4340-b574-b539dca36dc5","Type":"ContainerDied","Data":"05c21094e866ea8e21eb8c2207596c714882034e9a50a2e92cdc7d68eb184c9d"} Nov 03 16:22:54.686864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.686843 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" Nov 03 16:22:54.687012 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.686864 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh" event={"ID":"cb899c93-ba2a-4a6f-98ef-5c8cdabefa76","Type":"ContainerDied","Data":"beb44a0f5f7f82f50142e280afefb0159fba18bb6fe670231e3087d844c7d65f"} Nov 03 16:22:54.693927 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.693912 6319 scope.go:122] "RemoveContainer" containerID="8d36690bee9fae901faa4e7e2dec9319f6dfdd15aaf9d5bbb58d27463cf62275" Nov 03 16:22:54.702430 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.702414 6319 scope.go:122] "RemoveContainer" containerID="9fdc3327c8ffddb7bc47dcda2167b8ba5978350118fdaa4b8061fd65b05963a1" Nov 03 16:22:54.741780 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.741736 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-c4b666674-sbcxm" podStartSLOduration=2.259957715 podStartE2EDuration="2.741723676s" totalImagesPullingTime="481.765961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:54.738064343 +0000 UTC m=+1178.249432213" watchObservedRunningTime="2026-11-03 16:22:54.741723676 +0000 UTC m=+1178.253091538" Nov 03 16:22:54.782105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.782079 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x"] Nov 03 16:22:54.785714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.785694 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-h9k8x"] Nov 03 16:22:54.807608 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.807582 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9"] Nov 03 16:22:54.817921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.817890 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-jdcc9"] Nov 03 16:22:54.844567 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.844537 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh"] Nov 03 16:22:54.862688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:54.862666 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b132e3-openshift-default-5cdbfd9cf7-dfjsh"] Nov 03 16:22:56.643227 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:56.643156 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" path="/var/lib/kubelet/pods/1375f130-dfc4-4340-b574-b539dca36dc5/volumes" Nov 03 16:22:56.643603 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:56.643590 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" path="/var/lib/kubelet/pods/94cc12de-8730-4db9-a515-5ecd5d85d28f/volumes" Nov 03 16:22:56.643973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:22:56.643961 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" path="/var/lib/kubelet/pods/cb899c93-ba2a-4a6f-98ef-5c8cdabefa76/volumes" Nov 03 16:22:57.761558 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:22:57.761524 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:23:05.698106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:05.698076 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:23:07.037937 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.037901 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn"] Nov 03 16:23:07.038376 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038361 6319 cpu_manager.go:414] "containerMap: removing container" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" Nov 03 16:23:07.038416 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038380 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" Nov 03 16:23:07.038416 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038396 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" Nov 03 16:23:07.038416 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038401 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" Nov 03 16:23:07.038511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038419 6319 cpu_manager.go:414] "containerMap: removing container" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" Nov 03 16:23:07.038511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038424 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" Nov 03 16:23:07.038511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038441 6319 cpu_manager.go:414] "containerMap: removing container" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" Nov 03 16:23:07.038511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038446 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" Nov 03 16:23:07.038629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038517 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb899c93-ba2a-4a6f-98ef-5c8cdabefa76" containerName="istio-proxy" Nov 03 16:23:07.038629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038529 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1375f130-dfc4-4340-b574-b539dca36dc5" containerName="istio-proxy" Nov 03 16:23:07.038629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038542 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="94cc12de-8730-4db9-a515-5ecd5d85d28f" containerName="istio-proxy" Nov 03 16:23:07.038629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038550 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="d380c991-e43f-48a7-aa95-d4bd93207880" containerName="istio-proxy" Nov 03 16:23:07.038629 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.038556 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2f2cbd-6d19-4bc4-91e6-bd343bd5ecc1" containerName="istio-proxy" Nov 03 16:23:07.041625 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.041601 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.054886 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.054855 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn"] Nov 03 16:23:07.122313 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122280 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122313 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122316 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122523 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122340 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122523 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122424 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzwx\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122523 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122449 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122523 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122503 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122568 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122634 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.122722 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.122666 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.223799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223764 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223807 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223827 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223847 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223876 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzwx\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223891 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224020 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.223915 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224029 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224058 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224370 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224298 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224499 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224354 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224499 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224420 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224499 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224467 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.224859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.224842 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.226402 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.226380 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.226545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.226526 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.236513 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.236490 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.236709 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.236693 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzwx\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-z8vxn\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.360409 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.360334 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:07.640892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.640834 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn"] Nov 03 16:23:07.643208 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:23:07.643181 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc42cb7_799b_492d_b4cf_a99c4d099ecd.slice/crio-7da1627d71c47416c356feeb5bcdacb8cd5e2e71012db110aa60d8ac04490117 WatchSource:0}: Error finding container 7da1627d71c47416c356feeb5bcdacb8cd5e2e71012db110aa60d8ac04490117: Status 404 returned error can't find the container with id 7da1627d71c47416c356feeb5bcdacb8cd5e2e71012db110aa60d8ac04490117 Nov 03 16:23:07.645348 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.645318 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:23:07.645419 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.645398 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:23:07.645473 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.645425 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:23:07.764973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.764939 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" event={"ID":"7fc42cb7-799b-492d-b4cf-a99c4d099ecd","Type":"ContainerStarted","Data":"e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d"} Nov 03 16:23:07.765169 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:07.765018 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" event={"ID":"7fc42cb7-799b-492d-b4cf-a99c4d099ecd","Type":"ContainerStarted","Data":"7da1627d71c47416c356feeb5bcdacb8cd5e2e71012db110aa60d8ac04490117"} Nov 03 16:23:08.361350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:08.361307 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:08.362939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:08.362918 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:08.363027 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:08.362965 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:09.361271 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:09.361235 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:09.361453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:09.361292 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:10.360973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:10.360933 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:10.361171 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:10.361033 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:11.361630 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:11.361592 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:11.362053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:11.361645 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:12.361327 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:12.361292 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:12.361491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:12.361358 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:13.361378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:13.361347 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:13.361793 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:13.361400 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:14.361185 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:14.361152 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:23:14.361369 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:14.361217 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 16:23:16.361707 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:16.361667 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:23:16.362391 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:16.362334 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:23:17.364333 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:17.364290 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:23:17.364951 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:17.364921 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:23:18.406151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:18.406115 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:23:18.406606 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:18.406187 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:23:18.728434 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:18.728405 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:18.728737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:18.728706 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:18.729385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:18.729366 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:23:18.756623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:18.756577 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podStartSLOduration=11.756563290999999 podStartE2EDuration="11.756563291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:07.841857018 +0000 UTC m=+1191.353224886" watchObservedRunningTime="2026-11-03 16:23:18.756563291 +0000 UTC m=+1202.267931158" Nov 03 16:23:21.223165 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:21.223133 6319 scope.go:122] "RemoveContainer" containerID="af1d0ede3f52a84b68763dc1164e04821651f4b65fbd786164c2b61f1a7b78c7" Nov 03 16:23:24.923685 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:24.923655 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4b666674-sbcxm"] Nov 03 16:23:24.924087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:24.923855 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-c4b666674-sbcxm" podUID="761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" containerName="manager" containerID="cri-o://74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c" gracePeriod=30 Nov 03 16:23:25.178459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.178400 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:23:25.324202 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.324167 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww\" (UniqueName: \"kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww\") pod \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " Nov 03 16:23:25.324392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.324262 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs\") pod \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " Nov 03 16:23:25.324392 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.324315 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert\") pod \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\" (UID: \"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a\") " Nov 03 16:23:25.326598 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.326564 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww" pod "761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" (UID: "761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a"). InnerVolumeSpecName "kube-api-access-f9xww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:23:25.326598 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.326584 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs" pod "761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" (UID: "761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:23:25.326735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.326608 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert" pod "761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" (UID: "761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:23:25.425933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.425901 6319 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-webhook-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:23:25.425933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.425929 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xww\" (UniqueName: \"kubernetes.io/projected/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-kube-api-access-f9xww\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:23:25.425933 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.425941 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:23:25.904007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.903946 6319 generic.go:356] "Generic (PLEG): container finished" podID="761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" containerID="74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c" exitCode=0 Nov 03 16:23:25.904209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.904016 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-sbcxm" Nov 03 16:23:25.904209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.904029 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-sbcxm" event={"ID":"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a","Type":"ContainerDied","Data":"74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c"} Nov 03 16:23:25.904209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.904088 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-sbcxm" event={"ID":"761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a","Type":"ContainerDied","Data":"83a661dde19a9956cb45708ebf74a4f9f01e22694475d9024149d2327098e65c"} Nov 03 16:23:25.904209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.904124 6319 scope.go:122] "RemoveContainer" containerID="74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c" Nov 03 16:23:25.915162 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.915141 6319 scope.go:122] "RemoveContainer" containerID="74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c" Nov 03 16:23:25.915636 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:23:25.915566 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c\": container with ID starting with 74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c not found: ID does not exist" containerID="74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c" Nov 03 16:23:25.915636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.915602 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c"} err="failed to get container status \"74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c\": rpc error: code = NotFound desc = could not find container \"74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c\": container with ID starting with 74f1a2c438ddae6568ba7a283cda3180d0fa87f13a3889370b18df8269cb740c not found: ID does not exist" Nov 03 16:23:25.926634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.926612 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c4b666674-sbcxm"] Nov 03 16:23:25.931108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:25.931090 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c4b666674-sbcxm"] Nov 03 16:23:26.641469 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:26.641440 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" path="/var/lib/kubelet/pods/761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a/volumes" Nov 03 16:23:27.157307 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.157271 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c4b666674-b7hjc"] Nov 03 16:23:27.157695 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.157683 6319 cpu_manager.go:414] "containerMap: removing container" podUID="761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" containerName="manager" Nov 03 16:23:27.157750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.157697 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" containerName="manager" Nov 03 16:23:27.157785 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.157775 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="761f5b6f-86d1-4fbd-aff2-8a4fefe6b73a" containerName="manager" Nov 03 16:23:27.162220 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.162201 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.167541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.167515 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 16:23:27.167665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.167549 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 16:23:27.167665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.167560 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 16:23:27.167665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.167622 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7z7w6\"" Nov 03 16:23:27.171917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.171898 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4b666674-b7hjc"] Nov 03 16:23:27.344813 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.344776 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15c5062-14a5-4166-a0bb-f63d64a7b792-metrics-certs\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.345014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.344845 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pw9m\" (UniqueName: \"kubernetes.io/projected/e15c5062-14a5-4166-a0bb-f63d64a7b792-kube-api-access-2pw9m\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.345014 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.344911 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e15c5062-14a5-4166-a0bb-f63d64a7b792-webhook-cert\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.446371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.446301 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pw9m\" (UniqueName: \"kubernetes.io/projected/e15c5062-14a5-4166-a0bb-f63d64a7b792-kube-api-access-2pw9m\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.446371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.446339 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e15c5062-14a5-4166-a0bb-f63d64a7b792-webhook-cert\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.446543 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.446416 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15c5062-14a5-4166-a0bb-f63d64a7b792-metrics-certs\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.448941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.448910 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e15c5062-14a5-4166-a0bb-f63d64a7b792-metrics-certs\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.448941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.448934 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e15c5062-14a5-4166-a0bb-f63d64a7b792-webhook-cert\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.458727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.458708 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pw9m\" (UniqueName: \"kubernetes.io/projected/e15c5062-14a5-4166-a0bb-f63d64a7b792-kube-api-access-2pw9m\") pod \"maas-controller-c4b666674-b7hjc\" (UID: \"e15c5062-14a5-4166-a0bb-f63d64a7b792\") " pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.476439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.476422 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:27.751922 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.751897 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c4b666674-b7hjc"] Nov 03 16:23:27.754648 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:23:27.754621 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15c5062_14a5_4166_a0bb_f63d64a7b792.slice/crio-3ff0856168a4fa9dc9f522e954b12507e6865661e00c578fd810b4e26833f503 WatchSource:0}: Error finding container 3ff0856168a4fa9dc9f522e954b12507e6865661e00c578fd810b4e26833f503: Status 404 returned error can't find the container with id 3ff0856168a4fa9dc9f522e954b12507e6865661e00c578fd810b4e26833f503 Nov 03 16:23:27.763414 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:23:27.763389 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:23:27.916206 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:27.916168 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-b7hjc" event={"ID":"e15c5062-14a5-4166-a0bb-f63d64a7b792","Type":"ContainerStarted","Data":"3ff0856168a4fa9dc9f522e954b12507e6865661e00c578fd810b4e26833f503"} Nov 03 16:23:28.923239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:28.923209 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c4b666674-b7hjc" event={"ID":"e15c5062-14a5-4166-a0bb-f63d64a7b792","Type":"ContainerStarted","Data":"25771a517227c4b29c91143963db3ab39c3fda6359fb2be4af3a33a3fad04d94"} Nov 03 16:23:28.923635 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:28.923315 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:28.953919 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:28.953870 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-c4b666674-b7hjc" podStartSLOduration=1.571213691 podStartE2EDuration="1.953855142s" totalImagesPullingTime="382.641451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:23:28.952674863 +0000 UTC m=+1212.464042732" watchObservedRunningTime="2026-11-03 16:23:28.953855142 +0000 UTC m=+1212.465223008" Nov 03 16:23:39.929669 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:23:39.929636 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c4b666674-b7hjc" Nov 03 16:23:57.763864 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:23:57.763822 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:24:04.792144 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:24:04.792097 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:24:04.792756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:24:04.792723 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:24:27.761939 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:24:27.761853 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:24:46.295181 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:24:46.295126 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:24:46.296422 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:24:46.295958 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:24:46.421446 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:24:46.421410 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:24:46.421943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:24:46.421856 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:24:57.761458 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:24:57.761423 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:25:21.186009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:25:21.185119 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:25:21.186009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:25:21.185195 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:25:27.763134 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:25:27.763096 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:25:57.762006 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:25:57.761904 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:26:27.763733 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:26:27.763699 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:26:31.298212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:26:31.298128 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:26:31.299635 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:26:31.299587 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:26:31.421223 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:26:31.421168 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:26:31.421560 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:26:31.421512 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:26:57.845008 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:26:57.843197 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:27:01.291148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:01.291098 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:27:01.291792 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:01.291758 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:27:01.446766 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:01.446730 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:27:01.447972 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:01.447936 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:27:08.023254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:08.023222 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4"] Nov 03 16:27:08.029025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:08.029002 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789180a8-gnjl4"] Nov 03 16:27:08.641765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:27:08.641736 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34bbf58b-bc2c-4e52-9bbb-5ced5699f15d" path="/var/lib/kubelet/pods/34bbf58b-bc2c-4e52-9bbb-5ced5699f15d/volumes" Nov 03 16:27:27.763856 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:27:27.763777 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:27:57.761221 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:27:57.761181 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:28:21.401106 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:28:21.401074 6319 scope.go:122] "RemoveContainer" containerID="78d5edaedaa71236d4764425dcabf95eca32c0d9fccd20af904cb9843342f015" Nov 03 16:28:27.762198 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:28:27.762162 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:28:57.761958 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:28:57.761878 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:29:27.761458 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:29:27.761422 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:29:57.762417 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:29:57.762382 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:30:27.762356 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:30:27.762267 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:30:57.762022 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:30:57.761961 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:31:27.762496 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:31:27.762458 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:31:57.762779 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:31:57.762696 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:32:27.761391 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:32:27.761346 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:32:57.761728 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:32:57.761683 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:33:07.794915 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.794882 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t"] Nov 03 16:33:07.795477 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.795143 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" containerID="cri-o://46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1" gracePeriod=30 Nov 03 16:33:07.799558 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.799531 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn"] Nov 03 16:33:07.799777 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.799730 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" containerID="cri-o://e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d" gracePeriod=30 Nov 03 16:33:07.801013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.800995 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc"] Nov 03 16:33:07.801172 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.801153 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" containerID="cri-o://1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857" gracePeriod=30 Nov 03 16:33:07.802210 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.802187 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt"] Nov 03 16:33:07.802516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:07.802383 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" containerID="cri-o://ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587" gracePeriod=30 Nov 03 16:33:13.075759 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.075730 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:33:13.102038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.102008 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:33:13.141556 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.141522 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:33:13.149939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.149913 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:33:13.162080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.162047 6319 generic.go:356] "Generic (PLEG): container finished" podID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerID="e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d" exitCode=0 Nov 03 16:33:13.162268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.162097 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" Nov 03 16:33:13.162268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.162135 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" event={"ID":"7fc42cb7-799b-492d-b4cf-a99c4d099ecd","Type":"ContainerDied","Data":"e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d"} Nov 03 16:33:13.162268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.162186 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn" event={"ID":"7fc42cb7-799b-492d-b4cf-a99c4d099ecd","Type":"ContainerDied","Data":"7da1627d71c47416c356feeb5bcdacb8cd5e2e71012db110aa60d8ac04490117"} Nov 03 16:33:13.162268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.162206 6319 scope.go:122] "RemoveContainer" containerID="e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d" Nov 03 16:33:13.163976 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.163952 6319 generic.go:356] "Generic (PLEG): container finished" podID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerID="ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587" exitCode=0 Nov 03 16:33:13.164130 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.164019 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" Nov 03 16:33:13.164130 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.164053 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" event={"ID":"95e59cb1-9640-498e-a9b6-912e9f7465f5","Type":"ContainerDied","Data":"ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587"} Nov 03 16:33:13.164130 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.164093 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt" event={"ID":"95e59cb1-9640-498e-a9b6-912e9f7465f5","Type":"ContainerDied","Data":"d6a2790d7c9fad9dc3baefdc6eb58674bfdec18e791d552e2cbbb7eb6d6eb720"} Nov 03 16:33:13.165505 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.165482 6319 generic.go:356] "Generic (PLEG): container finished" podID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerID="46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1" exitCode=0 Nov 03 16:33:13.165610 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.165535 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" Nov 03 16:33:13.165610 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.165553 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" event={"ID":"fddde5f3-5f73-46af-b9d0-220a2c328c00","Type":"ContainerDied","Data":"46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1"} Nov 03 16:33:13.165610 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.165596 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t" event={"ID":"fddde5f3-5f73-46af-b9d0-220a2c328c00","Type":"ContainerDied","Data":"3ecae862faeb9727ab5e89653873f3b0742ccbe7feadb4e84d9ae601735386f8"} Nov 03 16:33:13.167038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.167019 6319 generic.go:356] "Generic (PLEG): container finished" podID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerID="1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857" exitCode=0 Nov 03 16:33:13.167110 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.167047 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" event={"ID":"4db585f2-62f8-4472-b652-c1c154ac97cf","Type":"ContainerDied","Data":"1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857"} Nov 03 16:33:13.167110 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.167062 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" Nov 03 16:33:13.167110 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.167096 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc" event={"ID":"4db585f2-62f8-4472-b652-c1c154ac97cf","Type":"ContainerDied","Data":"c34b0fe9918a2dd0503ee05b0f2a74700d822f14f0aff39aa379a994a6617349"} Nov 03 16:33:13.172331 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.172308 6319 scope.go:122] "RemoveContainer" containerID="e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d" Nov 03 16:33:13.172743 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:33:13.172700 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d\": container with ID starting with e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d not found: ID does not exist" containerID="e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d" Nov 03 16:33:13.172853 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.172736 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d"} err="failed to get container status \"e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d\": rpc error: code = NotFound desc = could not find container \"e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d\": container with ID starting with e1fad7d620b9c81b615699e317cc20eb94b55bfd4ab7338902b30b7c1e4f8f5d not found: ID does not exist" Nov 03 16:33:13.172853 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.172788 6319 scope.go:122] "RemoveContainer" containerID="ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587" Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.180894 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.180933 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.180975 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181027 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181075 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181109 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181137 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181162 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181195 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181222 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181248 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181291 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181325 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181365 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181393 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181429 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181483 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181512 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181550 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181578 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.181612 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.182681 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.184044 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.185736 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token\") pod \"fddde5f3-5f73-46af-b9d0-220a2c328c00\" (UID: \"fddde5f3-5f73-46af-b9d0-220a2c328c00\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.185791 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.185847 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.185880 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo\") pod \"4db585f2-62f8-4472-b652-c1c154ac97cf\" (UID: \"4db585f2-62f8-4472-b652-c1c154ac97cf\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.185946 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx\") pod \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\" (UID: \"7fc42cb7-799b-492d-b4cf-a99c4d099ecd\") " Nov 03 16:33:13.188921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.186456 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.192141 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.191634 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.192141 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.191670 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:33:13.192141 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.191885 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.193040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.192146 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.193040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.192180 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:13.193040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.192373 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.193040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.192691 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.193040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.192975 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:13.193344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.193208 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.193484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.193451 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:13.193931 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.193721 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.194145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.193730 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.194145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.194117 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.194301 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.194256 6319 scope.go:122] "RemoveContainer" containerID="ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587" Nov 03 16:33:13.194575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.194552 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.194674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.194580 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.194674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.194619 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.194793 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.194676 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.195167 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:33:13.195007 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587\": container with ID starting with ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587 not found: ID does not exist" containerID="ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587" Nov 03 16:33:13.195167 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.195049 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587"} err="failed to get container status \"ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587\": rpc error: code = NotFound desc = could not find container \"ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587\": container with ID starting with ea63bca16a9a6e75bb8f48c6f3da902c21db67551f7828989c06f96792c23587 not found: ID does not exist" Nov 03 16:33:13.195167 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.195075 6319 scope.go:122] "RemoveContainer" containerID="46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1" Nov 03 16:33:13.196170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.196079 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.196170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.196146 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.196170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.196166 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.197410 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.197347 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "kube-api-access-mg5k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.199549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.199511 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "kube-api-access-mmzwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.199655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.199552 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo" pod "4db585f2-62f8-4472-b652-c1c154ac97cf" (UID: "4db585f2-62f8-4472-b652-c1c154ac97cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:33:13.199655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.199555 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.199655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.199573 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp" pod "fddde5f3-5f73-46af-b9d0-220a2c328c00" (UID: "fddde5f3-5f73-46af-b9d0-220a2c328c00"). InnerVolumeSpecName "kube-api-access-8ttpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.199840 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.199678 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo" pod "7fc42cb7-799b-492d-b4cf-a99c4d099ecd" (UID: "7fc42cb7-799b-492d-b4cf-a99c4d099ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:33:13.205936 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.205915 6319 scope.go:122] "RemoveContainer" containerID="46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1" Nov 03 16:33:13.206423 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:33:13.206403 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1\": container with ID starting with 46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1 not found: ID does not exist" containerID="46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1" Nov 03 16:33:13.206480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.206433 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1"} err="failed to get container status \"46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1\": rpc error: code = NotFound desc = could not find container \"46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1\": container with ID starting with 46f4a7c82b4149f57d4402210b493feb76c2de00af37d013820290d61f6126a1 not found: ID does not exist" Nov 03 16:33:13.206480 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.206454 6319 scope.go:122] "RemoveContainer" containerID="1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857" Nov 03 16:33:13.217040 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.217016 6319 scope.go:122] "RemoveContainer" containerID="1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857" Nov 03 16:33:13.217355 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:33:13.217332 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857\": container with ID starting with 1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857 not found: ID does not exist" containerID="1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857" Nov 03 16:33:13.217453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.217364 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857"} err="failed to get container status \"1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857\": rpc error: code = NotFound desc = could not find container \"1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857\": container with ID starting with 1d43760723660fc263ef6285e5946ad0c0e7e18847fed8d16a4fd995a9f67857 not found: ID does not exist" Nov 03 16:33:13.286968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.286940 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287036 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287071 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287137 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287177 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287200 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287241 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287252 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.287509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287290 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287335 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn\") pod \"95e59cb1-9640-498e-a9b6-912e9f7465f5\" (UID: \"95e59cb1-9640-498e-a9b6-912e9f7465f5\") " Nov 03 16:33:13.287509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287457 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.287509 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287490 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287537 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287671 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287692 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287702 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287714 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287728 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287740 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287749 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmzwx\" (UniqueName: \"kubernetes.io/projected/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-kube-api-access-mmzwx\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287758 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287791 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287806 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e59cb1-9640-498e-a9b6-912e9f7465f5-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.287804 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287814 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db585f2-62f8-4472-b652-c1c154ac97cf-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287826 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287832 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287845 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5k8\" (UniqueName: \"kubernetes.io/projected/4db585f2-62f8-4472-b652-c1c154ac97cf-kube-api-access-mg5k8\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287864 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddde5f3-5f73-46af-b9d0-220a2c328c00-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287877 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287893 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287905 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ttpp\" (UniqueName: \"kubernetes.io/projected/fddde5f3-5f73-46af-b9d0-220a2c328c00-kube-api-access-8ttpp\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287918 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287932 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287945 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287956 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.287970 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288010 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288024 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288039 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288052 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db585f2-62f8-4472-b652-c1c154ac97cf-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288066 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288080 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc42cb7-799b-492d-b4cf-a99c4d099ecd-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288139 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.288282 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.288263 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddde5f3-5f73-46af-b9d0-220a2c328c00-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.289741 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.289711 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "kube-api-access-jdpmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.289861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.289733 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:13.290105 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.290071 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:33:13.290204 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.290120 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo" pod "95e59cb1-9640-498e-a9b6-912e9f7465f5" (UID: "95e59cb1-9640-498e-a9b6-912e9f7465f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:33:13.389403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.389324 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.389403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.389353 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.389403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.389364 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.389403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.389374 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e59cb1-9640-498e-a9b6-912e9f7465f5-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.389403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.389384 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdpmn\" (UniqueName: \"kubernetes.io/projected/95e59cb1-9640-498e-a9b6-912e9f7465f5-kube-api-access-jdpmn\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:33:13.494341 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.494308 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt"] Nov 03 16:33:13.499771 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.499744 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-qpwkt"] Nov 03 16:33:13.512576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.512553 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn"] Nov 03 16:33:13.518663 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.518636 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-z8vxn"] Nov 03 16:33:13.531698 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.531671 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc"] Nov 03 16:33:13.536305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.536280 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wrnjc"] Nov 03 16:33:13.548597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.548577 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t"] Nov 03 16:33:13.555684 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:13.555661 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-tbr8t"] Nov 03 16:33:14.640962 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:14.640868 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" path="/var/lib/kubelet/pods/4db585f2-62f8-4472-b652-c1c154ac97cf/volumes" Nov 03 16:33:14.641510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:14.641489 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" path="/var/lib/kubelet/pods/7fc42cb7-799b-492d-b4cf-a99c4d099ecd/volumes" Nov 03 16:33:14.641941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:14.641922 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" path="/var/lib/kubelet/pods/95e59cb1-9640-498e-a9b6-912e9f7465f5/volumes" Nov 03 16:33:14.642699 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:14.642677 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" path="/var/lib/kubelet/pods/fddde5f3-5f73-46af-b9d0-220a2c328c00/volumes" Nov 03 16:33:22.948171 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948138 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl"] Nov 03 16:33:22.948579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948545 6319 cpu_manager.go:414] "containerMap: removing container" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" Nov 03 16:33:22.948579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948557 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" Nov 03 16:33:22.948579 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948576 6319 cpu_manager.go:414] "containerMap: removing container" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948582 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948599 6319 cpu_manager.go:414] "containerMap: removing container" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948605 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948627 6319 cpu_manager.go:414] "containerMap: removing container" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948632 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948693 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db585f2-62f8-4472-b652-c1c154ac97cf" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948704 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e59cb1-9640-498e-a9b6-912e9f7465f5" containerName="istio-proxy" Nov 03 16:33:22.948714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948713 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddde5f3-5f73-46af-b9d0-220a2c328c00" containerName="istio-proxy" Nov 03 16:33:22.948999 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.948720 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc42cb7-799b-492d-b4cf-a99c4d099ecd" containerName="istio-proxy" Nov 03 16:33:22.951917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.951901 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:22.966205 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.966184 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ba8f54-openshift-default-dockercfg-4tblh\"" Nov 03 16:33:22.974860 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:22.974839 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl"] Nov 03 16:33:23.080463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080430 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7fj7\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080478 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080500 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080610 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080745 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080651 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080745 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080673 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080792 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080845 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080816 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.080845 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.080832 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181487 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181448 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181501 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181540 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181604 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181632 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181655 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181713 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7fj7\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181764 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.181843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181797 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.182112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.181901 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.182112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.182048 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.182112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.182090 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.182239 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.182183 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.182697 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.182666 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.184018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.183977 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.184424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.184406 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.194744 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.194719 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7fj7\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.194879 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.194815 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token\") pod \"e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.264880 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.264848 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:23.551125 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.551100 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl"] Nov 03 16:33:23.552160 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:33:23.552135 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e988408_ec8d_45de_935b_ed4ed175da25.slice/crio-8be864ebda229d33e23d7f5d4ed24bbdd806e9977a0e651e7d872273a5c946bb WatchSource:0}: Error finding container 8be864ebda229d33e23d7f5d4ed24bbdd806e9977a0e651e7d872273a5c946bb: Status 404 returned error can't find the container with id 8be864ebda229d33e23d7f5d4ed24bbdd806e9977a0e651e7d872273a5c946bb Nov 03 16:33:23.554193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.554161 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:23.554262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.554223 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:23.554262 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:23.554253 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:24.222341 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:24.222307 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" event={"ID":"1e988408-ec8d-45de-935b-ed4ed175da25","Type":"ContainerStarted","Data":"2eab6123294f676dfdd555c4d5ca8ddb875bd5d0406d3b273b9ea1f7a2eafc88"} Nov 03 16:33:24.222715 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:24.222397 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" event={"ID":"1e988408-ec8d-45de-935b-ed4ed175da25","Type":"ContainerStarted","Data":"8be864ebda229d33e23d7f5d4ed24bbdd806e9977a0e651e7d872273a5c946bb"} Nov 03 16:33:24.248514 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:24.248441 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podStartSLOduration=2.248428034 podStartE2EDuration="2.248428034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:24.246778997 +0000 UTC m=+1807.758146866" watchObservedRunningTime="2026-11-03 16:33:24.248428034 +0000 UTC m=+1807.759795901" Nov 03 16:33:24.265965 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:24.265936 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:24.271647 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:24.271624 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:25.227252 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:25.227220 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:25.228358 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:25.228337 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:27.767193 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:33:27.766957 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:33:27.843446 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.843411 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9"] Nov 03 16:33:27.846976 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.846959 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:27.855467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.855442 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ba8f54\"" Nov 03 16:33:27.857006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.856960 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c9eeec4f-dockercfg-pmpxx\"" Nov 03 16:33:27.858456 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.858432 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9"] Nov 03 16:33:27.885454 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.885423 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh"] Nov 03 16:33:27.888946 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.888921 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:27.900019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:27.899959 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh"] Nov 03 16:33:28.029403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.029301 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fvl\" (UniqueName: \"kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl\") pod \"payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.029403 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.029371 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2mn\" (UniqueName: \"kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn\") pod \"payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.029609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.029477 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.029609 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.029538 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.130411 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.130363 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fvl\" (UniqueName: \"kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl\") pod \"payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.130624 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.130421 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2mn\" (UniqueName: \"kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn\") pod \"payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.130624 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.130466 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.130624 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.130517 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.131278 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.131255 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.131390 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.131254 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.154885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.154853 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fvl\" (UniqueName: \"kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl\") pod \"payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.155079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.155009 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2mn\" (UniqueName: \"kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn\") pod \"payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.161651 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.161618 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:28.205092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.205063 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:28.474908 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.474883 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9"] Nov 03 16:33:28.477649 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:33:28.477622 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb393f99a_df55_4df6_8012_80512124b80b.slice/crio-20e2ea91ff7f7a7f8ecb29bc6dd16f4a6024254e590767a162226c3e5b3fe7e9 WatchSource:0}: Error finding container 20e2ea91ff7f7a7f8ecb29bc6dd16f4a6024254e590767a162226c3e5b3fe7e9: Status 404 returned error can't find the container with id 20e2ea91ff7f7a7f8ecb29bc6dd16f4a6024254e590767a162226c3e5b3fe7e9 Nov 03 16:33:28.511612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:28.511589 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh"] Nov 03 16:33:29.253612 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.253572 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" event={"ID":"7ba4861d-1655-4e43-af6e-119194c557b9","Type":"ContainerStarted","Data":"c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a"} Nov 03 16:33:29.254123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.253637 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" event={"ID":"7ba4861d-1655-4e43-af6e-119194c557b9","Type":"ContainerStarted","Data":"b827bf5735110f4e1e2d535e4a3345753977f37e2ad893cd560830b0ccd9f4fb"} Nov 03 16:33:29.254123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.253839 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:29.255379 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.255357 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" event={"ID":"b393f99a-df55-4df6-8012-80512124b80b","Type":"ContainerStarted","Data":"fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2"} Nov 03 16:33:29.255487 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.255391 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" event={"ID":"b393f99a-df55-4df6-8012-80512124b80b","Type":"ContainerStarted","Data":"20e2ea91ff7f7a7f8ecb29bc6dd16f4a6024254e590767a162226c3e5b3fe7e9"} Nov 03 16:33:29.255541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.255494 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:29.276460 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.276414 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" podStartSLOduration=2.2764015300000002 podStartE2EDuration="2.27640153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:29.275083053 +0000 UTC m=+1812.786450922" watchObservedRunningTime="2026-11-03 16:33:29.27640153 +0000 UTC m=+1812.787769397" Nov 03 16:33:29.298866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:29.298828 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" podStartSLOduration=2.2988168 podStartE2EDuration="2.2988168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:29.29880688 +0000 UTC m=+1812.810174748" watchObservedRunningTime="2026-11-03 16:33:29.2988168 +0000 UTC m=+1812.810184665" Nov 03 16:33:37.916004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:37.915952 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq"] Nov 03 16:33:37.920007 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:37.919960 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:37.936634 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:37.936602 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq"] Nov 03 16:33:38.001235 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.001199 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s22x6\"" Nov 03 16:33:38.027803 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.027764 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a875f2-059f-4327-a602-f421a3b2024b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.027837 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.027897 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.027919 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a875f2-059f-4327-a602-f421a3b2024b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028022 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.027956 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dxn\" (UniqueName: \"kubernetes.io/projected/60a875f2-059f-4327-a602-f421a3b2024b-kube-api-access-c5dxn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028213 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.028056 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028213 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.028103 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028213 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.028135 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a875f2-059f-4327-a602-f421a3b2024b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.028341 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.028240 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129121 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129078 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a875f2-059f-4327-a602-f421a3b2024b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129142 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129165 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129182 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a875f2-059f-4327-a602-f421a3b2024b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129201 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dxn\" (UniqueName: \"kubernetes.io/projected/60a875f2-059f-4327-a602-f421a3b2024b-kube-api-access-c5dxn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129220 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129254 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129284 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a875f2-059f-4327-a602-f421a3b2024b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129349 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129729 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129628 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.129817 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.129767 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.130043 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.130017 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.130145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.130105 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.130209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.130190 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a875f2-059f-4327-a602-f421a3b2024b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.132023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.132004 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a875f2-059f-4327-a602-f421a3b2024b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.132502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.132481 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a875f2-059f-4327-a602-f421a3b2024b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.152091 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.152060 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a875f2-059f-4327-a602-f421a3b2024b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.152245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.152219 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dxn\" (UniqueName: \"kubernetes.io/projected/60a875f2-059f-4327-a602-f421a3b2024b-kube-api-access-c5dxn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5lcmq\" (UID: \"60a875f2-059f-4327-a602-f421a3b2024b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.244069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.244030 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:38.538261 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.538230 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq"] Nov 03 16:33:38.540925 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:33:38.540896 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a875f2_059f_4327_a602_f421a3b2024b.slice/crio-4fafb59fe831061506ff45d2948c45a521859b5f505ac47c6b7604b87488b974 WatchSource:0}: Error finding container 4fafb59fe831061506ff45d2948c45a521859b5f505ac47c6b7604b87488b974: Status 404 returned error can't find the container with id 4fafb59fe831061506ff45d2948c45a521859b5f505ac47c6b7604b87488b974 Nov 03 16:33:38.543632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.543583 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:38.543721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.543682 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:38.543778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:38.543732 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:39.307495 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:39.307464 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" event={"ID":"60a875f2-059f-4327-a602-f421a3b2024b","Type":"ContainerStarted","Data":"11743f3c51d52b11f0f7ffb318e48df22dff0b09b8a206958e91bfa9b0d5c9c7"} Nov 03 16:33:39.307903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:39.307519 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" event={"ID":"60a875f2-059f-4327-a602-f421a3b2024b","Type":"ContainerStarted","Data":"4fafb59fe831061506ff45d2948c45a521859b5f505ac47c6b7604b87488b974"} Nov 03 16:33:39.337026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:39.336310 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podStartSLOduration=2.336289305 podStartE2EDuration="2.336289305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:39.33321628 +0000 UTC m=+1822.844584149" watchObservedRunningTime="2026-11-03 16:33:39.336289305 +0000 UTC m=+1822.847657168" Nov 03 16:33:40.244240 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.244195 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:40.245698 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.245667 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:40.245829 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.245718 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:40.263427 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.263396 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:33:40.263607 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.263469 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:33:40.764735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.764017 6319 patch_prober.go:29] interesting pod/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:57758->10.134.0.90:15021: read: connection reset by peer" start-of-body= Nov 03 16:33:40.764735 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:40.764085 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:57758->10.134.0.90:15021: read: connection reset by peer" Nov 03 16:33:41.245345 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.245306 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:41.245526 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.245366 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:41.350973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.350941 6319 generic.go:356] "Generic (PLEG): container finished" podID="1e988408-ec8d-45de-935b-ed4ed175da25" containerID="2eab6123294f676dfdd555c4d5ca8ddb875bd5d0406d3b273b9ea1f7a2eafc88" exitCode=0 Nov 03 16:33:41.351189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.351026 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" event={"ID":"1e988408-ec8d-45de-935b-ed4ed175da25","Type":"ContainerDied","Data":"2eab6123294f676dfdd555c4d5ca8ddb875bd5d0406d3b273b9ea1f7a2eafc88"} Nov 03 16:33:41.351631 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.351611 6319 scope.go:122] "RemoveContainer" containerID="2eab6123294f676dfdd555c4d5ca8ddb875bd5d0406d3b273b9ea1f7a2eafc88" Nov 03 16:33:41.352843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.352817 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:41.352903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.352891 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:41.352940 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:41.352926 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:33:42.249009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:42.248421 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:42.249009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:42.248483 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:42.367032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:42.366354 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" event={"ID":"1e988408-ec8d-45de-935b-ed4ed175da25","Type":"ContainerStarted","Data":"41ad0982cde1e38908a1adcb6fe6c9d9ad3f05f300acfd6c291080e10a66dcfd"} Nov 03 16:33:43.249010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:43.248409 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:43.249010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:43.248491 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:43.266099 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:43.266059 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:43.266273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:43.266237 6319 patch_prober.go:29] interesting pod/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:33:43.266329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:43.266279 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:33:44.250011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:44.245375 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:44.250011 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:44.245437 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:44.265236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:44.265209 6319 patch_prober.go:29] interesting pod/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:33:44.265436 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:44.265418 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:33:45.245347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:45.245309 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:45.245538 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:45.245361 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:45.266094 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:45.266061 6319 patch_prober.go:29] interesting pod/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:33:45.266467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:45.266108 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:33:46.245583 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:46.245539 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:33:46.245799 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:46.245616 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" podUID="60a875f2-059f-4327-a602-f421a3b2024b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:33:46.266115 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:46.266081 6319 patch_prober.go:29] interesting pod/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:33:46.266488 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:46.266133 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:33:47.265330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:47.265290 6319 patch_prober.go:29] interesting pod/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:33:47.265511 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:47.265352 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:33:47.694887 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:47.694802 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:47.696292 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:47.696266 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:47.697076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:47.697054 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5lcmq" Nov 03 16:33:48.269347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:48.269309 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:48.269693 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:48.269662 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:48.270504 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:33:48.270490 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:33:57.766484 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:33:57.766443 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:34:27.763107 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:34:27.763064 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:34:30.245627 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:30.245589 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh"] Nov 03 16:34:30.263966 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:30.263880 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9"] Nov 03 16:34:35.246679 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:35.246633 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" containerName="payload-processing" containerID="cri-o://c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a" gracePeriod=25 Nov 03 16:34:35.264725 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:35.264675 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" podUID="b393f99a-df55-4df6-8012-80512124b80b" containerName="payload-pre-processing" containerID="cri-o://fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2" gracePeriod=25 Nov 03 16:34:38.533233 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:38.533203 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl"] Nov 03 16:34:38.533685 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:38.533418 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" containerID="cri-o://41ad0982cde1e38908a1adcb6fe6c9d9ad3f05f300acfd6c291080e10a66dcfd" gracePeriod=30 Nov 03 16:34:40.261974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:40.261938 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" podUID="b393f99a-df55-4df6-8012-80512124b80b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:34:40.261974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:40.261955 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:34:43.669805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.669771 6319 generic.go:356] "Generic (PLEG): container finished" podID="1e988408-ec8d-45de-935b-ed4ed175da25" containerID="41ad0982cde1e38908a1adcb6fe6c9d9ad3f05f300acfd6c291080e10a66dcfd" exitCode=0 Nov 03 16:34:43.670350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.669846 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" event={"ID":"1e988408-ec8d-45de-935b-ed4ed175da25","Type":"ContainerDied","Data":"41ad0982cde1e38908a1adcb6fe6c9d9ad3f05f300acfd6c291080e10a66dcfd"} Nov 03 16:34:43.670350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.669913 6319 scope.go:122] "RemoveContainer" containerID="2eab6123294f676dfdd555c4d5ca8ddb875bd5d0406d3b273b9ea1f7a2eafc88" Nov 03 16:34:43.783547 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.783519 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:34:43.841005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.840909 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.840942 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841005 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.840995 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841065 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841100 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841125 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841160 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841195 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841251 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841230 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket\") pod \"1e988408-ec8d-45de-935b-ed4ed175da25\" (UID: \"1e988408-ec8d-45de-935b-ed4ed175da25\") " Nov 03 16:34:43.841537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841263 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:43.841537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841343 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:43.841670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841529 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:43.841670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841569 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:43.841670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841607 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:43.841670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841654 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.842136 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.841674 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.843340 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.843319 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:43.843500 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.843476 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:34:43.843557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.843534 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "kube-api-access-k7fj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:43.843798 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.843780 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token" pod "1e988408-ec8d-45de-935b-ed4ed175da25" (UID: "1e988408-ec8d-45de-935b-ed4ed175da25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:43.943087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943055 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7fj7\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-kube-api-access-k7fj7\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.943087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943082 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e988408-ec8d-45de-935b-ed4ed175da25-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.943087 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943092 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e988408-ec8d-45de-935b-ed4ed175da25-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.943311 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943101 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e988408-ec8d-45de-935b-ed4ed175da25-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.943311 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943110 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.943311 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943118 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:43.943311 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:43.943126 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e988408-ec8d-45de-935b-ed4ed175da25-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:44.676349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:44.676322 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" Nov 03 16:34:44.676696 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:44.676329 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl" event={"ID":"1e988408-ec8d-45de-935b-ed4ed175da25","Type":"ContainerDied","Data":"8be864ebda229d33e23d7f5d4ed24bbdd806e9977a0e651e7d872273a5c946bb"} Nov 03 16:34:44.676696 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:44.676448 6319 scope.go:122] "RemoveContainer" containerID="41ad0982cde1e38908a1adcb6fe6c9d9ad3f05f300acfd6c291080e10a66dcfd" Nov 03 16:34:44.697146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:44.697122 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl"] Nov 03 16:34:44.703790 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:44.703770 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ba8f54-openshift-default-5d8f8459492h6xl"] Nov 03 16:34:46.641725 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:46.641694 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" path="/var/lib/kubelet/pods/1e988408-ec8d-45de-935b-ed4ed175da25/volumes" Nov 03 16:34:50.262353 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:50.262321 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:34:50.262353 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:50.262340 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" podUID="b393f99a-df55-4df6-8012-80512124b80b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:34:55.513199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.513132 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:34:55.540871 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.540836 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:34:55.653117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.653090 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume\") pod \"7ba4861d-1655-4e43-af6e-119194c557b9\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " Nov 03 16:34:55.653303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.653145 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume\") pod \"b393f99a-df55-4df6-8012-80512124b80b\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " Nov 03 16:34:55.653303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.653173 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl\" (UniqueName: \"kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl\") pod \"7ba4861d-1655-4e43-af6e-119194c557b9\" (UID: \"7ba4861d-1655-4e43-af6e-119194c557b9\") " Nov 03 16:34:55.653303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.653195 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn\" (UniqueName: \"kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn\") pod \"b393f99a-df55-4df6-8012-80512124b80b\" (UID: \"b393f99a-df55-4df6-8012-80512124b80b\") " Nov 03 16:34:55.653533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.653509 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume" pod "7ba4861d-1655-4e43-af6e-119194c557b9" (UID: "7ba4861d-1655-4e43-af6e-119194c557b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:55.653630 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.653609 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume" pod "b393f99a-df55-4df6-8012-80512124b80b" (UID: "b393f99a-df55-4df6-8012-80512124b80b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:55.655486 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.655456 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn" pod "b393f99a-df55-4df6-8012-80512124b80b" (UID: "b393f99a-df55-4df6-8012-80512124b80b"). InnerVolumeSpecName "kube-api-access-7p2mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:55.655577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.655463 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl" pod "7ba4861d-1655-4e43-af6e-119194c557b9" (UID: "7ba4861d-1655-4e43-af6e-119194c557b9"). InnerVolumeSpecName "kube-api-access-t4fvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:55.726517 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.726486 6319 generic.go:356] "Generic (PLEG): container finished" podID="7ba4861d-1655-4e43-af6e-119194c557b9" containerID="c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a" exitCode=0 Nov 03 16:34:55.726668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.726572 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" event={"ID":"7ba4861d-1655-4e43-af6e-119194c557b9","Type":"ContainerDied","Data":"c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a"} Nov 03 16:34:55.726668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.726587 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" Nov 03 16:34:55.726668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.726632 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh" event={"ID":"7ba4861d-1655-4e43-af6e-119194c557b9","Type":"ContainerDied","Data":"b827bf5735110f4e1e2d535e4a3345753977f37e2ad893cd560830b0ccd9f4fb"} Nov 03 16:34:55.726668 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.726661 6319 scope.go:122] "RemoveContainer" containerID="c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a" Nov 03 16:34:55.727962 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.727933 6319 generic.go:356] "Generic (PLEG): container finished" podID="b393f99a-df55-4df6-8012-80512124b80b" containerID="fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2" exitCode=0 Nov 03 16:34:55.728098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.727973 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" Nov 03 16:34:55.728098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.728019 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" event={"ID":"b393f99a-df55-4df6-8012-80512124b80b","Type":"ContainerDied","Data":"fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2"} Nov 03 16:34:55.728098 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.728056 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9" event={"ID":"b393f99a-df55-4df6-8012-80512124b80b","Type":"ContainerDied","Data":"20e2ea91ff7f7a7f8ecb29bc6dd16f4a6024254e590767a162226c3e5b3fe7e9"} Nov 03 16:34:55.735654 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.735548 6319 scope.go:122] "RemoveContainer" containerID="c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a" Nov 03 16:34:55.735872 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:34:55.735834 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a\": container with ID starting with c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a not found: ID does not exist" containerID="c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a" Nov 03 16:34:55.736006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.735876 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a"} err="failed to get container status \"c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a\": rpc error: code = NotFound desc = could not find container \"c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a\": container with ID starting with c03a67375099685d0184bc4b17217c57d1b248a83079ee001cc1ea88e8e39a7a not found: ID does not exist" Nov 03 16:34:55.736006 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.735898 6319 scope.go:122] "RemoveContainer" containerID="fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2" Nov 03 16:34:55.746041 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.745998 6319 scope.go:122] "RemoveContainer" containerID="fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2" Nov 03 16:34:55.746299 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:34:55.746269 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2\": container with ID starting with fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2 not found: ID does not exist" containerID="fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2" Nov 03 16:34:55.746393 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.746304 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2"} err="failed to get container status \"fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2\": rpc error: code = NotFound desc = could not find container \"fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2\": container with ID starting with fdfa56f1274817eb0ef2afa254a7849d7cb4e1a4c79030e3da62ba3a6c0cedc2 not found: ID does not exist" Nov 03 16:34:55.753917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.753894 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b393f99a-df55-4df6-8012-80512124b80b-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:55.754037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.753920 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4fvl\" (UniqueName: \"kubernetes.io/projected/7ba4861d-1655-4e43-af6e-119194c557b9-kube-api-access-t4fvl\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:55.754037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.753931 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2mn\" (UniqueName: \"kubernetes.io/projected/b393f99a-df55-4df6-8012-80512124b80b-kube-api-access-7p2mn\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:55.754037 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.753940 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba4861d-1655-4e43-af6e-119194c557b9-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:34:55.754943 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.754923 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9"] Nov 03 16:34:55.760289 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.760269 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ba8f54-86f65sghq9"] Nov 03 16:34:55.771838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.771783 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh"] Nov 03 16:34:55.778438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:55.778419 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ba8f54-6875cb8d58vjmh"] Nov 03 16:34:56.641199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:56.641125 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" path="/var/lib/kubelet/pods/7ba4861d-1655-4e43-af6e-119194c557b9/volumes" Nov 03 16:34:56.641546 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:34:56.641477 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b393f99a-df55-4df6-8012-80512124b80b" path="/var/lib/kubelet/pods/b393f99a-df55-4df6-8012-80512124b80b/volumes" Nov 03 16:34:57.762547 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:34:57.762511 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:35:08.095757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.095716 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54"] Nov 03 16:35:08.096408 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096362 6319 cpu_manager.go:414] "containerMap: removing container" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" containerName="payload-processing" Nov 03 16:35:08.096408 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096382 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" containerName="payload-processing" Nov 03 16:35:08.096408 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096407 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" Nov 03 16:35:08.096596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096416 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" Nov 03 16:35:08.096596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096430 6319 cpu_manager.go:414] "containerMap: removing container" podUID="b393f99a-df55-4df6-8012-80512124b80b" containerName="payload-pre-processing" Nov 03 16:35:08.096596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096438 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b393f99a-df55-4df6-8012-80512124b80b" containerName="payload-pre-processing" Nov 03 16:35:08.096596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096561 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" Nov 03 16:35:08.096596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096579 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba4861d-1655-4e43-af6e-119194c557b9" containerName="payload-processing" Nov 03 16:35:08.096596 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096592 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" Nov 03 16:35:08.096896 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.096602 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="b393f99a-df55-4df6-8012-80512124b80b" containerName="payload-pre-processing" Nov 03 16:35:08.099807 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.099786 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.111640 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.111617 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54"] Nov 03 16:35:08.266899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.266869 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.266899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.266903 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c35b1f9-9955-47e5-9864-6ff55045a21a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267142 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.266924 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267142 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.267016 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267142 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.267058 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nbg\" (UniqueName: \"kubernetes.io/projected/8c35b1f9-9955-47e5-9864-6ff55045a21a-kube-api-access-k4nbg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267142 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.267127 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.267181 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267286 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.267223 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.267353 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.267286 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368366 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368416 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c35b1f9-9955-47e5-9864-6ff55045a21a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368444 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368469 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368500 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nbg\" (UniqueName: \"kubernetes.io/projected/8c35b1f9-9955-47e5-9864-6ff55045a21a-kube-api-access-k4nbg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368550 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368630 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368678 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.368764 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368731 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.369131 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.368907 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.369131 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.369005 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.369247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.369131 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.369247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.369147 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.369321 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.369243 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c35b1f9-9955-47e5-9864-6ff55045a21a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.371088 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.371065 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.371203 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.371182 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.386075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.386046 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nbg\" (UniqueName: \"kubernetes.io/projected/8c35b1f9-9955-47e5-9864-6ff55045a21a-kube-api-access-k4nbg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.386187 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.386120 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c35b1f9-9955-47e5-9864-6ff55045a21a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-ldj54\" (UID: \"8c35b1f9-9955-47e5-9864-6ff55045a21a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.415864 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.415839 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:08.703363 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.703336 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54"] Nov 03 16:35:08.705654 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:35:08.705622 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c35b1f9_9955_47e5_9864_6ff55045a21a.slice/crio-52f4c657beaf42857d07e7c6287d16d44c26d53d7b8fd9e88a2cf30a5d53243c WatchSource:0}: Error finding container 52f4c657beaf42857d07e7c6287d16d44c26d53d7b8fd9e88a2cf30a5d53243c: Status 404 returned error can't find the container with id 52f4c657beaf42857d07e7c6287d16d44c26d53d7b8fd9e88a2cf30a5d53243c Nov 03 16:35:08.707770 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.707729 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:08.707867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.707802 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:08.707867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.707830 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:08.789544 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.789510 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" event={"ID":"8c35b1f9-9955-47e5-9864-6ff55045a21a","Type":"ContainerStarted","Data":"401e423f11d0bf0cf365635b2f1eee374d047f2105a28c589e985d2c482b1cf4"} Nov 03 16:35:08.789711 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.789573 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" event={"ID":"8c35b1f9-9955-47e5-9864-6ff55045a21a","Type":"ContainerStarted","Data":"52f4c657beaf42857d07e7c6287d16d44c26d53d7b8fd9e88a2cf30a5d53243c"} Nov 03 16:35:08.816081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:08.816025 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podStartSLOduration=0.816010063 podStartE2EDuration="816.010063ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:08.813901534 +0000 UTC m=+1912.325269404" watchObservedRunningTime="2026-11-03 16:35:08.816010063 +0000 UTC m=+1912.327377931" Nov 03 16:35:09.417050 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:09.417017 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:09.418564 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:09.418540 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:35:09.418682 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:09.418586 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:35:10.420159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:10.420124 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:10.798967 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:10.798937 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:10.799885 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:10.799865 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:35:23.071192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.071163 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl"] Nov 03 16:35:23.071623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.071578 6319 cpu_manager.go:414] "containerMap: removing container" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" Nov 03 16:35:23.071623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.071589 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e988408-ec8d-45de-935b-ed4ed175da25" containerName="istio-proxy" Nov 03 16:35:23.076151 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.076134 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.095201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.095168 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl"] Nov 03 16:35:23.103757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.103733 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.103895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.103761 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af06ee06-d105-42d0-8efc-77603192cc29-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.103895 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.103790 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9htl\" (UniqueName: \"kubernetes.io/projected/af06ee06-d105-42d0-8efc-77603192cc29-kube-api-access-x9htl\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.104018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.103929 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.104018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.103958 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.104120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.104036 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.104120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.104059 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.104187 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.104157 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af06ee06-d105-42d0-8efc-77603192cc29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.104231 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.104189 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af06ee06-d105-42d0-8efc-77603192cc29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.204924 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.204880 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9htl\" (UniqueName: \"kubernetes.io/projected/af06ee06-d105-42d0-8efc-77603192cc29-kube-api-access-x9htl\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.204932 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205182 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205118 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205221 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205194 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205267 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205224 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205274 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af06ee06-d105-42d0-8efc-77603192cc29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205345 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205307 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af06ee06-d105-42d0-8efc-77603192cc29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205362 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205385 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205378 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205479 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205441 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205522 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205493 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af06ee06-d105-42d0-8efc-77603192cc29-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205569 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205536 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205726 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.205907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.205889 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af06ee06-d105-42d0-8efc-77603192cc29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.207626 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.207604 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af06ee06-d105-42d0-8efc-77603192cc29-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.207909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.207893 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af06ee06-d105-42d0-8efc-77603192cc29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.217599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.217580 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af06ee06-d105-42d0-8efc-77603192cc29-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.217737 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.217720 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9htl\" (UniqueName: \"kubernetes.io/projected/af06ee06-d105-42d0-8efc-77603192cc29-kube-api-access-x9htl\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wlqrl\" (UID: \"af06ee06-d105-42d0-8efc-77603192cc29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.393345 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.393249 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:23.676184 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.676131 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl"] Nov 03 16:35:23.678372 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:35:23.678337 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf06ee06_d105_42d0_8efc_77603192cc29.slice/crio-e66f6da329650e503d223b38366ac3c2a936d9f62f5c0d7e7fa2b6842c4dbc34 WatchSource:0}: Error finding container e66f6da329650e503d223b38366ac3c2a936d9f62f5c0d7e7fa2b6842c4dbc34: Status 404 returned error can't find the container with id e66f6da329650e503d223b38366ac3c2a936d9f62f5c0d7e7fa2b6842c4dbc34 Nov 03 16:35:23.680672 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.680642 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:23.680749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.680706 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:23.680749 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.680736 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:23.867654 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.867616 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" event={"ID":"af06ee06-d105-42d0-8efc-77603192cc29","Type":"ContainerStarted","Data":"273584456fd9151bb659dd0ce97343a76c9c5f2902a2411b8ede747aff445030"} Nov 03 16:35:23.867867 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.867679 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" event={"ID":"af06ee06-d105-42d0-8efc-77603192cc29","Type":"ContainerStarted","Data":"e66f6da329650e503d223b38366ac3c2a936d9f62f5c0d7e7fa2b6842c4dbc34"} Nov 03 16:35:23.892915 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:23.892858 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podStartSLOduration=0.892843627 podStartE2EDuration="892.843627ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:23.892181611 +0000 UTC m=+1927.403549480" watchObservedRunningTime="2026-11-03 16:35:23.892843627 +0000 UTC m=+1927.404211495" Nov 03 16:35:24.394229 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:24.394194 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:24.395670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:24.395650 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:35:24.395726 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:24.395695 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:35:25.398058 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:25.398026 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:25.877615 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:25.877586 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:25.878536 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:35:25.878515 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:35:27.762433 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:35:27.762398 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:35:57.761480 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:35:57.761439 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:36:26.841189 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:26.841139 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:26.841756 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:26.841220 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:26.921201 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:26.921161 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:26.921378 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:26.921249 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:27.765614 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:36:27.765569 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:36:29.220230 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.220195 6319 generic.go:356] "Generic (PLEG): container finished" podID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerID="401e423f11d0bf0cf365635b2f1eee374d047f2105a28c589e985d2c482b1cf4" exitCode=0 Nov 03 16:36:29.220998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.220269 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" event={"ID":"8c35b1f9-9955-47e5-9864-6ff55045a21a","Type":"ContainerDied","Data":"401e423f11d0bf0cf365635b2f1eee374d047f2105a28c589e985d2c482b1cf4"} Nov 03 16:36:29.220998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.220697 6319 scope.go:122] "RemoveContainer" containerID="401e423f11d0bf0cf365635b2f1eee374d047f2105a28c589e985d2c482b1cf4" Nov 03 16:36:29.221954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.221921 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:29.222080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.222012 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:29.222080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.222053 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:29.222080 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.222050 6319 generic.go:356] "Generic (PLEG): container finished" podID="af06ee06-d105-42d0-8efc-77603192cc29" containerID="273584456fd9151bb659dd0ce97343a76c9c5f2902a2411b8ede747aff445030" exitCode=0 Nov 03 16:36:29.222197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.222122 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" event={"ID":"af06ee06-d105-42d0-8efc-77603192cc29","Type":"ContainerDied","Data":"273584456fd9151bb659dd0ce97343a76c9c5f2902a2411b8ede747aff445030"} Nov 03 16:36:29.222425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.222403 6319 scope.go:122] "RemoveContainer" containerID="273584456fd9151bb659dd0ce97343a76c9c5f2902a2411b8ede747aff445030" Nov 03 16:36:29.223662 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.223633 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:29.223743 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.223712 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:29.223795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:29.223762 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:30.264170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.264135 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" event={"ID":"8c35b1f9-9955-47e5-9864-6ff55045a21a","Type":"ContainerStarted","Data":"4d3782eef85d7c62f0cb964e95c9b769bbacb74f9eb89f9b9fde47f22c5769eb"} Nov 03 16:36:30.266368 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.266346 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" event={"ID":"af06ee06-d105-42d0-8efc-77603192cc29","Type":"ContainerStarted","Data":"e6ea30bcb65d5e92306a03ae90ab80a73e9ec62ccbba48c1682c52243f0cae68"} Nov 03 16:36:30.394145 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.394112 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:36:30.394460 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.394437 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:30.394534 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.394489 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:30.424466 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.424432 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:36:30.424843 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.424759 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:30.425026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:30.424975 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:31.394614 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:31.394572 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:31.395155 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:31.394633 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:31.419384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:31.419349 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:31.420058 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:31.420026 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:32.394042 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:32.394006 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:32.394232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:32.394061 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:32.416406 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:32.416372 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:32.417010 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:32.416956 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:33.394252 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:33.394217 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:33.394442 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:33.394271 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:33.420008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:33.416370 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:33.420008 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:33.416429 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:34.394470 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:34.394430 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:34.394663 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:34.394481 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:34.417584 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:34.417551 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:34.417879 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:34.417850 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:35.394038 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:35.393967 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:35.394453 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:35.394073 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:35.417198 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:35.417162 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:35.417364 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:35.417214 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:36.394302 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:36.394265 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:36:36.394712 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:36.394351 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:36:36.416268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:36.416236 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:36:36.416429 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:36.416299 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 16:36:38.417146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:38.417091 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:38.417659 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:38.417208 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:38.427815 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:38.427784 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:38.428111 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:38.428066 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:39.428616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:39.428565 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:39.429324 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:39.429282 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:39.460224 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:39.460179 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:39.460510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:39.460467 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:40.463530 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:40.463483 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:40.464108 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:40.463561 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:40.472139 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:40.472107 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:40.472347 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:40.472323 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:40.799977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:40.799888 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:36:40.877721 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:40.877688 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:36:41.474750 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:41.474713 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:41.475810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:41.475414 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:41.505172 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:41.505131 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-ldj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:41.505320 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:41.505238 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" podUID="8c35b1f9-9955-47e5-9864-6ff55045a21a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:42.410412 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:42.410384 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:36:42.411468 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:42.411444 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-ldj54" Nov 03 16:36:42.517188 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:42.517151 6319 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:36:42.517706 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:42.517678 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" podUID="af06ee06-d105-42d0-8efc-77603192cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:36:43.313957 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:43.313923 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:36:43.315009 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:43.314971 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wlqrl" Nov 03 16:36:57.762415 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:36:57.762374 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:36:58.744622 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.744585 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr"] Nov 03 16:36:58.747428 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.747380 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.762894 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.762870 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-13ba71-serving-cert\"" Nov 03 16:36:58.767704 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.767679 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr"] Nov 03 16:36:58.844100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.844069 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.844290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.844109 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j98fl\" (UniqueName: \"kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.844290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.844142 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.844290 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.844269 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.901932 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.901896 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr"] Nov 03 16:36:58.904312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.904295 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:58.918317 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.918280 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-13ba71\"" Nov 03 16:36:58.921263 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.921236 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr"] Nov 03 16:36:58.945860 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.945832 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.946039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.945868 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98fl\" (UniqueName: \"kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.946039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.945901 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.946162 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.946032 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.946614 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.946591 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.948476 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.948450 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.948731 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.948709 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:58.962919 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:58.962894 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98fl\" (UniqueName: \"kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl\") pod \"maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:59.053075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.049520 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.053075 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.049634 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d8z\" (UniqueName: \"kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z\") pod \"payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.064281 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.062957 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:36:59.150703 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.150672 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d8z\" (UniqueName: \"kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z\") pod \"payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.150870 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.150780 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.151437 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.151419 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.170032 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.169952 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d8z\" (UniqueName: \"kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z\") pod \"payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.215977 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.215945 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:36:59.368209 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.368187 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr"] Nov 03 16:36:59.370173 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:36:59.370145 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706eab12_b167_4999_8d13_36fbc7b9ae48.slice/crio-a5abd7d8349f4e564bc000caf7a58b538e6d258dd60d7ccc55048ebce9a35961 WatchSource:0}: Error finding container a5abd7d8349f4e564bc000caf7a58b538e6d258dd60d7ccc55048ebce9a35961: Status 404 returned error can't find the container with id a5abd7d8349f4e564bc000caf7a58b538e6d258dd60d7ccc55048ebce9a35961 Nov 03 16:36:59.371599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.371578 6319 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:36:59.431920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.431888 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" event={"ID":"706eab12-b167-4999-8d13-36fbc7b9ae48","Type":"ContainerStarted","Data":"a5abd7d8349f4e564bc000caf7a58b538e6d258dd60d7ccc55048ebce9a35961"} Nov 03 16:36:59.498149 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:36:59.498125 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr"] Nov 03 16:36:59.500332 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:36:59.500298 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0518dc47_7345_4cce_8454_5dc371541c86.slice/crio-b968e8226f02c30d87c7a918eca34b84fd08bb03d71fbcf61cd6cc32fe478bc4 WatchSource:0}: Error finding container b968e8226f02c30d87c7a918eca34b84fd08bb03d71fbcf61cd6cc32fe478bc4: Status 404 returned error can't find the container with id b968e8226f02c30d87c7a918eca34b84fd08bb03d71fbcf61cd6cc32fe478bc4 Nov 03 16:37:00.441960 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.441911 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" event={"ID":"0518dc47-7345-4cce-8454-5dc371541c86","Type":"ContainerStarted","Data":"5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4"} Nov 03 16:37:00.442479 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.442008 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" event={"ID":"0518dc47-7345-4cce-8454-5dc371541c86","Type":"ContainerStarted","Data":"b968e8226f02c30d87c7a918eca34b84fd08bb03d71fbcf61cd6cc32fe478bc4"} Nov 03 16:37:00.442479 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.442149 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:37:00.444296 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.444267 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" event={"ID":"706eab12-b167-4999-8d13-36fbc7b9ae48","Type":"ContainerStarted","Data":"d0310ae77f83e170c921c46471937969aa0fde606a3d3064b1cc6a9a4f18a2a4"} Nov 03 16:37:00.444907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.444889 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:37:00.466059 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.465964 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" podStartSLOduration=2.465946457 podStartE2EDuration="2.465946457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:00.462998841 +0000 UTC m=+2023.974366701" watchObservedRunningTime="2026-11-03 16:37:00.465946457 +0000 UTC m=+2023.977314326" Nov 03 16:37:00.495256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:00.495214 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" podStartSLOduration=2.123034396 podStartE2EDuration="2.495201158s" totalImagesPullingTime="372.166762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:00.494184907 +0000 UTC m=+2024.005552788" watchObservedRunningTime="2026-11-03 16:37:00.495201158 +0000 UTC m=+2024.006569026" Nov 03 16:37:07.462117 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:07.462085 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:37:11.453033 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:37:11.453000 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:37:27.762924 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:37:27.762880 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:37:57.762654 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:37:57.762569 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:38:06.121587 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.121555 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr"] Nov 03 16:38:06.375778 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.375679 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr"] Nov 03 16:38:06.376604 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.376559 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" podUID="706eab12-b167-4999-8d13-36fbc7b9ae48" containerName="maas-api" containerID="cri-o://d0310ae77f83e170c921c46471937969aa0fde606a3d3064b1cc6a9a4f18a2a4" gracePeriod=30 Nov 03 16:38:06.846838 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.846811 6319 generic.go:356] "Generic (PLEG): container finished" podID="706eab12-b167-4999-8d13-36fbc7b9ae48" containerID="d0310ae77f83e170c921c46471937969aa0fde606a3d3064b1cc6a9a4f18a2a4" exitCode=0 Nov 03 16:38:06.847016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.846890 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" event={"ID":"706eab12-b167-4999-8d13-36fbc7b9ae48","Type":"ContainerDied","Data":"d0310ae77f83e170c921c46471937969aa0fde606a3d3064b1cc6a9a4f18a2a4"} Nov 03 16:38:06.847016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.846941 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" event={"ID":"706eab12-b167-4999-8d13-36fbc7b9ae48","Type":"ContainerDied","Data":"a5abd7d8349f4e564bc000caf7a58b538e6d258dd60d7ccc55048ebce9a35961"} Nov 03 16:38:06.847016 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.846967 6319 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5abd7d8349f4e564bc000caf7a58b538e6d258dd60d7ccc55048ebce9a35961" Nov 03 16:38:06.850325 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.850305 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:38:06.924459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.924369 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls\") pod \"706eab12-b167-4999-8d13-36fbc7b9ae48\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " Nov 03 16:38:06.924459 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.924426 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs\") pod \"706eab12-b167-4999-8d13-36fbc7b9ae48\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " Nov 03 16:38:06.924646 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.924583 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca\") pod \"706eab12-b167-4999-8d13-36fbc7b9ae48\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " Nov 03 16:38:06.924690 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.924647 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl\" (UniqueName: \"kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl\") pod \"706eab12-b167-4999-8d13-36fbc7b9ae48\" (UID: \"706eab12-b167-4999-8d13-36fbc7b9ae48\") " Nov 03 16:38:06.925035 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.925003 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca" pod "706eab12-b167-4999-8d13-36fbc7b9ae48" (UID: "706eab12-b167-4999-8d13-36fbc7b9ae48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:06.926779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.926750 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls" pod "706eab12-b167-4999-8d13-36fbc7b9ae48" (UID: "706eab12-b167-4999-8d13-36fbc7b9ae48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:06.926920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.926783 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl" pod "706eab12-b167-4999-8d13-36fbc7b9ae48" (UID: "706eab12-b167-4999-8d13-36fbc7b9ae48"). InnerVolumeSpecName "kube-api-access-j98fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:06.926920 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:06.926843 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs" pod "706eab12-b167-4999-8d13-36fbc7b9ae48" (UID: "706eab12-b167-4999-8d13-36fbc7b9ae48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:07.026256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.026214 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706eab12-b167-4999-8d13-36fbc7b9ae48-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:07.026256 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.026248 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j98fl\" (UniqueName: \"kubernetes.io/projected/706eab12-b167-4999-8d13-36fbc7b9ae48-kube-api-access-j98fl\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:07.026510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.026263 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:07.026510 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.026280 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706eab12-b167-4999-8d13-36fbc7b9ae48-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:07.852531 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.852500 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr" Nov 03 16:38:07.881127 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.881098 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr"] Nov 03 16:38:07.888727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:07.888701 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-13ba71-59997bbc75-w74hr"] Nov 03 16:38:08.641146 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:08.641117 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706eab12-b167-4999-8d13-36fbc7b9ae48" path="/var/lib/kubelet/pods/706eab12-b167-4999-8d13-36fbc7b9ae48/volumes" Nov 03 16:38:11.122013 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:11.121919 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" podUID="0518dc47-7345-4cce-8454-5dc371541c86" containerName="payload-processing" containerID="cri-o://5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4" gracePeriod=25 Nov 03 16:38:11.451787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:11.451704 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" podUID="0518dc47-7345-4cce-8454-5dc371541c86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:38:21.471120 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:21.471075 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" podUID="0518dc47-7345-4cce-8454-5dc371541c86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:38:27.762209 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:38:27.762170 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:38:31.378424 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.378368 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:38:31.480640 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.480610 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z\" (UniqueName: \"kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z\") pod \"0518dc47-7345-4cce-8454-5dc371541c86\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " Nov 03 16:38:31.480849 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.480711 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume\") pod \"0518dc47-7345-4cce-8454-5dc371541c86\" (UID: \"0518dc47-7345-4cce-8454-5dc371541c86\") " Nov 03 16:38:31.481140 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.481114 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume" pod "0518dc47-7345-4cce-8454-5dc371541c86" (UID: "0518dc47-7345-4cce-8454-5dc371541c86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:31.482938 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.482913 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z" pod "0518dc47-7345-4cce-8454-5dc371541c86" (UID: "0518dc47-7345-4cce-8454-5dc371541c86"). InnerVolumeSpecName "kube-api-access-r5d8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:31.582175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.582141 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5d8z\" (UniqueName: \"kubernetes.io/projected/0518dc47-7345-4cce-8454-5dc371541c86-kube-api-access-r5d8z\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:31.582175 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:31.582176 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518dc47-7345-4cce-8454-5dc371541c86-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:38:32.000776 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.000741 6319 generic.go:356] "Generic (PLEG): container finished" podID="0518dc47-7345-4cce-8454-5dc371541c86" containerID="5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4" exitCode=0 Nov 03 16:38:32.000973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.000790 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" Nov 03 16:38:32.000973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.000833 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" event={"ID":"0518dc47-7345-4cce-8454-5dc371541c86","Type":"ContainerDied","Data":"5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4"} Nov 03 16:38:32.000973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.000890 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr" event={"ID":"0518dc47-7345-4cce-8454-5dc371541c86","Type":"ContainerDied","Data":"b968e8226f02c30d87c7a918eca34b84fd08bb03d71fbcf61cd6cc32fe478bc4"} Nov 03 16:38:32.000973 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.000915 6319 scope.go:122] "RemoveContainer" containerID="5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4" Nov 03 16:38:32.010170 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.010143 6319 scope.go:122] "RemoveContainer" containerID="5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4" Nov 03 16:38:32.010436 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:38:32.010418 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4\": container with ID starting with 5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4 not found: ID does not exist" containerID="5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4" Nov 03 16:38:32.010501 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.010446 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4"} err="failed to get container status \"5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4\": rpc error: code = NotFound desc = could not find container \"5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4\": container with ID starting with 5935cdba8b20483090c27df604d9d525a53e5756cd0ef3a5d6b99027420e5fd4 not found: ID does not exist" Nov 03 16:38:32.023841 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.023815 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr"] Nov 03 16:38:32.027961 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.027934 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-13ba71-849cd7979xdvsr"] Nov 03 16:38:32.641599 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:38:32.641559 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0518dc47-7345-4cce-8454-5dc371541c86" path="/var/lib/kubelet/pods/0518dc47-7345-4cce-8454-5dc371541c86/volumes" Nov 03 16:38:57.761858 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:38:57.761816 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:39:27.761599 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:39:27.761508 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:39:40.930708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.930671 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz"] Nov 03 16:39:40.931193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.931176 6319 cpu_manager.go:414] "containerMap: removing container" podUID="706eab12-b167-4999-8d13-36fbc7b9ae48" containerName="maas-api" Nov 03 16:39:40.931245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.931195 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706eab12-b167-4999-8d13-36fbc7b9ae48" containerName="maas-api" Nov 03 16:39:40.931245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.931210 6319 cpu_manager.go:414] "containerMap: removing container" podUID="0518dc47-7345-4cce-8454-5dc371541c86" containerName="payload-processing" Nov 03 16:39:40.931245 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.931216 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0518dc47-7345-4cce-8454-5dc371541c86" containerName="payload-processing" Nov 03 16:39:40.931352 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.931290 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="706eab12-b167-4999-8d13-36fbc7b9ae48" containerName="maas-api" Nov 03 16:39:40.931352 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.931299 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="0518dc47-7345-4cce-8454-5dc371541c86" containerName="payload-processing" Nov 03 16:39:40.934473 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.934456 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:40.969015 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:40.968964 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz"] Nov 03 16:39:41.066566 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.066519 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkl7\" (UniqueName: \"kubernetes.io/projected/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-kube-api-access-mnkl7\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.066761 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.066576 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-openshift-service-ca\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.066812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.066756 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-maas-api-tls\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.066812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.066783 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-metrics-certs\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.167848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.167803 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkl7\" (UniqueName: \"kubernetes.io/projected/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-kube-api-access-mnkl7\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.167848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.167854 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-openshift-service-ca\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.168193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.167962 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-maas-api-tls\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.168193 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.168017 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-metrics-certs\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.168812 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.168735 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-openshift-service-ca\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.170676 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.170654 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-maas-api-tls\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.170794 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.170777 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-metrics-certs\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.180697 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.180667 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkl7\" (UniqueName: \"kubernetes.io/projected/e098d0c5-4c68-48bf-8651-8b2bbe5612f4-kube-api-access-mnkl7\") pod \"maas-api-55668dcc6b-c4ncz\" (UID: \"e098d0c5-4c68-48bf-8651-8b2bbe5612f4\") " pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.247855 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.247817 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:41.534884 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:41.534854 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz"] Nov 03 16:39:41.536392 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:39:41.536354 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode098d0c5_4c68_48bf_8651_8b2bbe5612f4.slice/crio-3a2f7e5b7ce33748bb55e95f1e6c239901f91a22ce542eb4c00c018934a3578a WatchSource:0}: Error finding container 3a2f7e5b7ce33748bb55e95f1e6c239901f91a22ce542eb4c00c018934a3578a: Status 404 returned error can't find the container with id 3a2f7e5b7ce33748bb55e95f1e6c239901f91a22ce542eb4c00c018934a3578a Nov 03 16:39:42.485926 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:42.485890 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" event={"ID":"e098d0c5-4c68-48bf-8651-8b2bbe5612f4","Type":"ContainerStarted","Data":"281ba4b501f7e514c1842e63630efa48bd720b7aceb88692df04a8c6c5417e74"} Nov 03 16:39:42.486475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:42.485954 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" event={"ID":"e098d0c5-4c68-48bf-8651-8b2bbe5612f4","Type":"ContainerStarted","Data":"3a2f7e5b7ce33748bb55e95f1e6c239901f91a22ce542eb4c00c018934a3578a"} Nov 03 16:39:42.486475 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:42.486045 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:42.509255 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:42.509200 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" podStartSLOduration=2.075137769 podStartE2EDuration="2.509184816s" totalImagesPullingTime="434.047047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:42.507264749 +0000 UTC m=+2186.018632617" watchObservedRunningTime="2026-11-03 16:39:42.509184816 +0000 UTC m=+2186.020552684" Nov 03 16:39:48.497287 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:48.497261 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55668dcc6b-c4ncz" Nov 03 16:39:52.549796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.549762 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt"] Nov 03 16:39:52.553575 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.553557 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.571198 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.571160 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt"] Nov 03 16:39:52.598931 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.598894 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g"] Nov 03 16:39:52.602497 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.602476 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.613616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.613591 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g"] Nov 03 16:39:52.618508 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.618483 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-56d83be3-dockercfg-vprv8\"" Nov 03 16:39:52.634443 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.634411 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-56d83be3\"" Nov 03 16:39:52.700541 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.692709 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7lt\" (UniqueName: \"kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt\") pod \"payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.700921 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.700889 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.701148 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.701129 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5vzt\" (UniqueName: \"kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt\") pod \"payload-processing-e2e-ait-56d83be3-fcf784559-8t22g\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.701315 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.701287 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume\") pod \"payload-processing-e2e-ait-56d83be3-fcf784559-8t22g\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.802577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.802486 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.802577 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.802539 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5vzt\" (UniqueName: \"kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt\") pod \"payload-processing-e2e-ait-56d83be3-fcf784559-8t22g\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.802822 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.802627 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume\") pod \"payload-processing-e2e-ait-56d83be3-fcf784559-8t22g\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.802822 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.802703 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7lt\" (UniqueName: \"kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt\") pod \"payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.803330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.803306 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.803463 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.803442 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume\") pod \"payload-processing-e2e-ait-56d83be3-fcf784559-8t22g\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.821426 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.821396 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7lt\" (UniqueName: \"kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt\") pod \"payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.821567 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.821395 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5vzt\" (UniqueName: \"kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt\") pod \"payload-processing-e2e-ait-56d83be3-fcf784559-8t22g\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:52.867321 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.867290 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:52.916796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:52.916763 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:53.172938 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.172906 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt"] Nov 03 16:39:53.175400 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:39:53.175369 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec695ee3_fd37_4d77_8785_b9463e092818.slice/crio-8de6d9da96d16bdcff33bc2fce0605b005720a5fbf7efc48884ecc4f4863b42a WatchSource:0}: Error finding container 8de6d9da96d16bdcff33bc2fce0605b005720a5fbf7efc48884ecc4f4863b42a: Status 404 returned error can't find the container with id 8de6d9da96d16bdcff33bc2fce0605b005720a5fbf7efc48884ecc4f4863b42a Nov 03 16:39:53.207765 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.207742 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g"] Nov 03 16:39:53.213193 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:39:53.213166 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d1b8d6_2c01_4952_80b6_d9b8c44b520b.slice/crio-453ed984a2880e11d17de019c35297b6880ae5333c4f41d7c4cd2c9b1946319e WatchSource:0}: Error finding container 453ed984a2880e11d17de019c35297b6880ae5333c4f41d7c4cd2c9b1946319e: Status 404 returned error can't find the container with id 453ed984a2880e11d17de019c35297b6880ae5333c4f41d7c4cd2c9b1946319e Nov 03 16:39:53.548210 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.548173 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" event={"ID":"ec695ee3-fd37-4d77-8785-b9463e092818","Type":"ContainerStarted","Data":"4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6"} Nov 03 16:39:53.548431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.548274 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:39:53.548431 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.548295 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" event={"ID":"ec695ee3-fd37-4d77-8785-b9463e092818","Type":"ContainerStarted","Data":"8de6d9da96d16bdcff33bc2fce0605b005720a5fbf7efc48884ecc4f4863b42a"} Nov 03 16:39:53.549648 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.549622 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" event={"ID":"13d1b8d6-2c01-4952-80b6-d9b8c44b520b","Type":"ContainerStarted","Data":"7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa"} Nov 03 16:39:53.549781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.549660 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" event={"ID":"13d1b8d6-2c01-4952-80b6-d9b8c44b520b","Type":"ContainerStarted","Data":"453ed984a2880e11d17de019c35297b6880ae5333c4f41d7c4cd2c9b1946319e"} Nov 03 16:39:53.549781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.549695 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:39:53.570846 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.570790 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" podStartSLOduration=1.5707733259999999 podStartE2EDuration="1.570773326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:53.570048851 +0000 UTC m=+2197.081416718" watchObservedRunningTime="2026-11-03 16:39:53.570773326 +0000 UTC m=+2197.082141195" Nov 03 16:39:53.591670 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:53.591598 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" podStartSLOduration=1.591584724 podStartE2EDuration="1.591584724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:53.588846737 +0000 UTC m=+2197.100214610" watchObservedRunningTime="2026-11-03 16:39:53.591584724 +0000 UTC m=+2197.102952593" Nov 03 16:39:55.792297 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:55.792255 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q"] Nov 03 16:39:55.792787 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:55.792520 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" podUID="0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" containerName="maas-api" containerID="cri-o://466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3" gracePeriod=30 Nov 03 16:39:57.065421 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.065396 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:39:57.249271 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.249236 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca\") pod \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " Nov 03 16:39:57.249456 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.249296 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6\" (UniqueName: \"kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6\") pod \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " Nov 03 16:39:57.249456 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.249365 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs\") pod \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " Nov 03 16:39:57.249456 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.249430 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls\") pod \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\" (UID: \"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7\") " Nov 03 16:39:57.249713 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.249687 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca" pod "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" (UID: "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:57.251688 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.251655 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6" pod "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" (UID: "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7"). InnerVolumeSpecName "kube-api-access-spnv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:57.251820 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.251756 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls" pod "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" (UID: "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:57.251820 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.251774 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs" pod "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" (UID: "0b72f7e3-78ff-4111-b4c9-a24dfade5ba7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:57.350401 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.350363 6319 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-metrics-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:57.350401 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.350397 6319 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-maas-api-tls\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:57.350620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.350413 6319 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-openshift-service-ca\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:57.350620 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.350429 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnv6\" (UniqueName: \"kubernetes.io/projected/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7-kube-api-access-spnv6\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:39:57.569968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.569871 6319 generic.go:356] "Generic (PLEG): container finished" podID="0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" containerID="466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3" exitCode=0 Nov 03 16:39:57.569968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.569916 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" Nov 03 16:39:57.569968 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.569947 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" event={"ID":"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7","Type":"ContainerDied","Data":"466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3"} Nov 03 16:39:57.570265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.570023 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q" event={"ID":"0b72f7e3-78ff-4111-b4c9-a24dfade5ba7","Type":"ContainerDied","Data":"385234d122e0076088ab8868f73c284a0a8baa28d091cd119ce1e8075f6a1a2d"} Nov 03 16:39:57.570265 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.570053 6319 scope.go:122] "RemoveContainer" containerID="466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3" Nov 03 16:39:57.579926 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.579906 6319 scope.go:122] "RemoveContainer" containerID="466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3" Nov 03 16:39:57.580259 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:39:57.580240 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3\": container with ID starting with 466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3 not found: ID does not exist" containerID="466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3" Nov 03 16:39:57.580328 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.580268 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3"} err="failed to get container status \"466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3\": rpc error: code = NotFound desc = could not find container \"466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3\": container with ID starting with 466746330d882afc3970be24871036c3c4b620118b3effe7c528d1a2185fb8d3 not found: ID does not exist" Nov 03 16:39:57.593816 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.593782 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q"] Nov 03 16:39:57.602700 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:57.602678 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-87dc4db6-9tl8q"] Nov 03 16:39:57.763012 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:39:57.762962 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:39:58.641587 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:39:58.641556 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" path="/var/lib/kubelet/pods/0b72f7e3-78ff-4111-b4c9-a24dfade5ba7/volumes" Nov 03 16:40:04.557053 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:04.557025 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:40:04.557545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:04.557093 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:40:17.242250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:17.242219 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g"] Nov 03 16:40:17.281149 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:17.281113 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt"] Nov 03 16:40:22.243109 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.243030 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" podUID="13d1b8d6-2c01-4952-80b6-d9b8c44b520b" containerName="payload-processing" containerID="cri-o://7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa" gracePeriod=25 Nov 03 16:40:22.281952 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.281905 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" podUID="ec695ee3-fd37-4d77-8785-b9463e092818" containerName="payload-pre-processing" containerID="cri-o://4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6" gracePeriod=25 Nov 03 16:40:22.535744 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.535721 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:40:22.568442 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.568419 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:40:22.617129 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.617098 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt\" (UniqueName: \"kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt\") pod \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " Nov 03 16:40:22.617314 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.617191 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume\") pod \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\" (UID: \"13d1b8d6-2c01-4952-80b6-d9b8c44b520b\") " Nov 03 16:40:22.617314 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.617298 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume\") pod \"ec695ee3-fd37-4d77-8785-b9463e092818\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " Nov 03 16:40:22.617444 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.617331 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt\" (UniqueName: \"kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt\") pod \"ec695ee3-fd37-4d77-8785-b9463e092818\" (UID: \"ec695ee3-fd37-4d77-8785-b9463e092818\") " Nov 03 16:40:22.617601 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.617577 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume" pod "13d1b8d6-2c01-4952-80b6-d9b8c44b520b" (UID: "13d1b8d6-2c01-4952-80b6-d9b8c44b520b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:22.617687 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.617662 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume" pod "ec695ee3-fd37-4d77-8785-b9463e092818" (UID: "ec695ee3-fd37-4d77-8785-b9463e092818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:22.619439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.619413 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt" pod "13d1b8d6-2c01-4952-80b6-d9b8c44b520b" (UID: "13d1b8d6-2c01-4952-80b6-d9b8c44b520b"). InnerVolumeSpecName "kube-api-access-x5vzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:22.619548 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.619497 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt" pod "ec695ee3-fd37-4d77-8785-b9463e092818" (UID: "ec695ee3-fd37-4d77-8785-b9463e092818"). InnerVolumeSpecName "kube-api-access-nl7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:22.700309 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.700279 6319 generic.go:356] "Generic (PLEG): container finished" podID="13d1b8d6-2c01-4952-80b6-d9b8c44b520b" containerID="7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa" exitCode=0 Nov 03 16:40:22.700484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.700326 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" Nov 03 16:40:22.700484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.700360 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" event={"ID":"13d1b8d6-2c01-4952-80b6-d9b8c44b520b","Type":"ContainerDied","Data":"7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa"} Nov 03 16:40:22.700484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.700410 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g" event={"ID":"13d1b8d6-2c01-4952-80b6-d9b8c44b520b","Type":"ContainerDied","Data":"453ed984a2880e11d17de019c35297b6880ae5333c4f41d7c4cd2c9b1946319e"} Nov 03 16:40:22.700484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.700440 6319 scope.go:122] "RemoveContainer" containerID="7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa" Nov 03 16:40:22.701898 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.701872 6319 generic.go:356] "Generic (PLEG): container finished" podID="ec695ee3-fd37-4d77-8785-b9463e092818" containerID="4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6" exitCode=0 Nov 03 16:40:22.701898 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.701905 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" event={"ID":"ec695ee3-fd37-4d77-8785-b9463e092818","Type":"ContainerDied","Data":"4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6"} Nov 03 16:40:22.702100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.701920 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" Nov 03 16:40:22.702100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.701934 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt" event={"ID":"ec695ee3-fd37-4d77-8785-b9463e092818","Type":"ContainerDied","Data":"8de6d9da96d16bdcff33bc2fce0605b005720a5fbf7efc48884ecc4f4863b42a"} Nov 03 16:40:22.709805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.709788 6319 scope.go:122] "RemoveContainer" containerID="7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa" Nov 03 16:40:22.710134 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:40:22.710114 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa\": container with ID starting with 7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa not found: ID does not exist" containerID="7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa" Nov 03 16:40:22.710236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.710140 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa"} err="failed to get container status \"7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa\": rpc error: code = NotFound desc = could not find container \"7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa\": container with ID starting with 7a1f2213f7d0309c42bcfdcce9469b92d6c3bee34fc9483178ba2c92a1b474aa not found: ID does not exist" Nov 03 16:40:22.710236 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.710157 6319 scope.go:122] "RemoveContainer" containerID="4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6" Nov 03 16:40:22.718197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.718176 6319 scope.go:122] "RemoveContainer" containerID="4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6" Nov 03 16:40:22.718534 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:40:22.718510 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6\": container with ID starting with 4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6 not found: ID does not exist" containerID="4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6" Nov 03 16:40:22.718632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.718534 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:22.718632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.718541 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6"} err="failed to get container status \"4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6\": rpc error: code = NotFound desc = could not find container \"4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6\": container with ID starting with 4dabd369bbfe3b6a1763f82e4a54a2e89b28787f47cf5b641bf9de61c54573a6 not found: ID does not exist" Nov 03 16:40:22.718632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.718556 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec695ee3-fd37-4d77-8785-b9463e092818-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:22.718632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.718571 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl7lt\" (UniqueName: \"kubernetes.io/projected/ec695ee3-fd37-4d77-8785-b9463e092818-kube-api-access-nl7lt\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:22.718632 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.718585 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5vzt\" (UniqueName: \"kubernetes.io/projected/13d1b8d6-2c01-4952-80b6-d9b8c44b520b-kube-api-access-x5vzt\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:40:22.719837 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.719816 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g"] Nov 03 16:40:22.730703 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.730672 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56d83be3-fcf784559-8t22g"] Nov 03 16:40:22.737779 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.737745 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt"] Nov 03 16:40:22.745103 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:22.745086 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56d83be3-95bcfd959-8tfxt"] Nov 03 16:40:24.642241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:24.642209 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d1b8d6-2c01-4952-80b6-d9b8c44b520b" path="/var/lib/kubelet/pods/13d1b8d6-2c01-4952-80b6-d9b8c44b520b/volumes" Nov 03 16:40:24.642660 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:24.642558 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec695ee3-fd37-4d77-8785-b9463e092818" path="/var/lib/kubelet/pods/ec695ee3-fd37-4d77-8785-b9463e092818/volumes" Nov 03 16:40:25.240452 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.240418 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56"] Nov 03 16:40:25.241149 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241130 6319 cpu_manager.go:414] "containerMap: removing container" podUID="0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" containerName="maas-api" Nov 03 16:40:25.241199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241153 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" containerName="maas-api" Nov 03 16:40:25.241199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241187 6319 cpu_manager.go:414] "containerMap: removing container" podUID="ec695ee3-fd37-4d77-8785-b9463e092818" containerName="payload-pre-processing" Nov 03 16:40:25.241199 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241196 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec695ee3-fd37-4d77-8785-b9463e092818" containerName="payload-pre-processing" Nov 03 16:40:25.241303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241205 6319 cpu_manager.go:414] "containerMap: removing container" podUID="13d1b8d6-2c01-4952-80b6-d9b8c44b520b" containerName="payload-processing" Nov 03 16:40:25.241303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241248 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d1b8d6-2c01-4952-80b6-d9b8c44b520b" containerName="payload-processing" Nov 03 16:40:25.241377 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241366 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b72f7e3-78ff-4111-b4c9-a24dfade5ba7" containerName="maas-api" Nov 03 16:40:25.241420 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241383 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec695ee3-fd37-4d77-8785-b9463e092818" containerName="payload-pre-processing" Nov 03 16:40:25.241420 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.241393 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d1b8d6-2c01-4952-80b6-d9b8c44b520b" containerName="payload-processing" Nov 03 16:40:25.246212 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.246192 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.255019 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.255000 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-05bcf4-openshift-default-dockercfg-dftqx\"" Nov 03 16:40:25.256940 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.256917 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56"] Nov 03 16:40:25.344232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344198 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344235 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59m7\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344269 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344408 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344450 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344566 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344476 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344566 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344527 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344635 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344564 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.344635 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.344595 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.445739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.445701 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.445739 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.445739 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59m7\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446031 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.445789 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446031 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.445852 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446218 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446250 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446225 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446288 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446292 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446350 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446377 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446402 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.446831 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446729 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.447026 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.446971 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.447195 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.447176 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.449845 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.449550 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.456846 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.455738 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.466533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.466499 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59m7\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.469255 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.469218 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token\") pod \"e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.560181 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.560083 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:25.845112 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.845088 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56"] Nov 03 16:40:25.847891 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:40:25.847866 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaab4a89_6e52_4544_a1ce_825fcb7f2e85.slice/crio-d7450e3471adb36cb73878b715f841f3c8c2bc3b3ac1c899e0b287c0b663bf92 WatchSource:0}: Error finding container d7450e3471adb36cb73878b715f841f3c8c2bc3b3ac1c899e0b287c0b663bf92: Status 404 returned error can't find the container with id d7450e3471adb36cb73878b715f841f3c8c2bc3b3ac1c899e0b287c0b663bf92 Nov 03 16:40:25.850103 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.850068 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:25.850159 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.850149 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:25.850192 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:25.850184 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:26.726502 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:26.726462 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" event={"ID":"aaab4a89-6e52-4544-a1ce-825fcb7f2e85","Type":"ContainerStarted","Data":"3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69"} Nov 03 16:40:26.726691 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:26.726517 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" event={"ID":"aaab4a89-6e52-4544-a1ce-825fcb7f2e85","Type":"ContainerStarted","Data":"d7450e3471adb36cb73878b715f841f3c8c2bc3b3ac1c899e0b287c0b663bf92"} Nov 03 16:40:26.753449 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:26.753393 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podStartSLOduration=1.753378305 podStartE2EDuration="1.753378305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:26.752206682 +0000 UTC m=+2230.263574550" watchObservedRunningTime="2026-11-03 16:40:26.753378305 +0000 UTC m=+2230.264746233" Nov 03 16:40:27.560581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:27.560553 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:27.565674 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:27.565652 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:27.731231 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:27.731196 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:27.732237 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:27.732218 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:27.768640 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:40:27.768601 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:40:30.426247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.426214 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp"] Nov 03 16:40:30.430024 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.430008 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.435533 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.435509 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs"] Nov 03 16:40:30.439077 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.439059 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.443849 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.443829 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-05bcf4\"" Nov 03 16:40:30.445190 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.445172 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-05bcf4-dockercfg-rd42g\"" Nov 03 16:40:30.455187 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.455165 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp"] Nov 03 16:40:30.463314 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.463253 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs"] Nov 03 16:40:30.494299 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.494266 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk567\" (UniqueName: \"kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567\") pod \"payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.494484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.494325 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99ln\" (UniqueName: \"kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln\") pod \"payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.494484 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.494352 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.494655 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.494629 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.595907 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.595857 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.596101 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.595972 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk567\" (UniqueName: \"kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567\") pod \"payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.596101 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.596054 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ln\" (UniqueName: \"kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln\") pod \"payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.596101 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.596083 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.596789 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.596765 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.597065 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.597041 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.620664 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.619113 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99ln\" (UniqueName: \"kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln\") pod \"payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.627998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.627029 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk567\" (UniqueName: \"kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567\") pod \"payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:30.744296 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.744258 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:30.752434 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:30.752410 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:31.031288 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.031199 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp"] Nov 03 16:40:31.035269 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:40:31.035236 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f6517a_baa9_455b_baa7_0cf94a4e82a5.slice/crio-3e88232569bb26cc7180e9e2903c9a3d2d2f06681634ab8b664ce1c86edc0035 WatchSource:0}: Error finding container 3e88232569bb26cc7180e9e2903c9a3d2d2f06681634ab8b664ce1c86edc0035: Status 404 returned error can't find the container with id 3e88232569bb26cc7180e9e2903c9a3d2d2f06681634ab8b664ce1c86edc0035 Nov 03 16:40:31.061322 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.061288 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs"] Nov 03 16:40:31.064568 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:40:31.064530 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc834d700_cbe1_4ff6_99f1_93481ac6864f.slice/crio-40f26abe20c91ae8878fda1cdd6bf431099463887494f410a0fbcbc0922096b2 WatchSource:0}: Error finding container 40f26abe20c91ae8878fda1cdd6bf431099463887494f410a0fbcbc0922096b2: Status 404 returned error can't find the container with id 40f26abe20c91ae8878fda1cdd6bf431099463887494f410a0fbcbc0922096b2 Nov 03 16:40:31.752023 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.751956 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" event={"ID":"72f6517a-baa9-455b-baa7-0cf94a4e82a5","Type":"ContainerStarted","Data":"a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf"} Nov 03 16:40:31.752514 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.752051 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" event={"ID":"72f6517a-baa9-455b-baa7-0cf94a4e82a5","Type":"ContainerStarted","Data":"3e88232569bb26cc7180e9e2903c9a3d2d2f06681634ab8b664ce1c86edc0035"} Nov 03 16:40:31.752514 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.752080 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:31.753736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.753710 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" event={"ID":"c834d700-cbe1-4ff6-99f1-93481ac6864f","Type":"ContainerStarted","Data":"d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea"} Nov 03 16:40:31.753861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.753750 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" event={"ID":"c834d700-cbe1-4ff6-99f1-93481ac6864f","Type":"ContainerStarted","Data":"40f26abe20c91ae8878fda1cdd6bf431099463887494f410a0fbcbc0922096b2"} Nov 03 16:40:31.753861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.753783 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:31.807306 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.807249 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" podStartSLOduration=1.8072359 podStartE2EDuration="1.8072359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:31.805851522 +0000 UTC m=+2235.317219389" watchObservedRunningTime="2026-11-03 16:40:31.8072359 +0000 UTC m=+2235.318603768" Nov 03 16:40:31.857542 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:31.857483 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" podStartSLOduration=1.857469719 podStartE2EDuration="1.857469719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:31.856524256 +0000 UTC m=+2235.367892135" watchObservedRunningTime="2026-11-03 16:40:31.857469719 +0000 UTC m=+2235.368837587" Nov 03 16:40:42.732559 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.732518 6319 patch_prober.go:29] interesting pod/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 16:40:42.733039 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.732608 6319 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 16:40:42.762425 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.762401 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:40:42.762586 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.762463 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:40:42.811439 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.811410 6319 generic.go:356] "Generic (PLEG): container finished" podID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerID="3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69" exitCode=0 Nov 03 16:40:42.811616 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.811487 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" event={"ID":"aaab4a89-6e52-4544-a1ce-825fcb7f2e85","Type":"ContainerDied","Data":"3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69"} Nov 03 16:40:42.811917 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.811893 6319 scope.go:122] "RemoveContainer" containerID="3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69" Nov 03 16:40:42.813247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.813217 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:42.813349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.813291 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:42.813349 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:42.813324 6319 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:43.842079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:43.842043 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" event={"ID":"aaab4a89-6e52-4544-a1ce-825fcb7f2e85","Type":"ContainerStarted","Data":"614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0"} Nov 03 16:40:44.561123 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:44.561092 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:44.561365 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:44.561344 6319 patch_prober.go:29] interesting pod/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 16:40:44.561418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:44.561390 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 16:40:45.561440 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:45.561402 6319 patch_prober.go:29] interesting pod/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 16:40:45.561860 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:45.561473 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 16:40:46.560796 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:46.560759 6319 patch_prober.go:29] interesting pod/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 16:40:46.561004 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:46.560813 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 16:40:47.560806 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:47.560773 6319 patch_prober.go:29] interesting pod/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 16:40:47.561215 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:47.560831 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 16:40:48.560808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:48.560765 6319 patch_prober.go:29] interesting pod/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 16:40:48.561350 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:48.560832 6319 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 16:40:49.564443 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:49.564414 6319 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:49.564882 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:49.564784 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:49.565391 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:40:49.565372 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:40:57.761767 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:40:57.761685 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:41:27.763457 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:41:27.763420 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:41:57.762073 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:41:57.762036 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:42:14.411861 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:14.411829 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs"] Nov 03 16:42:14.431371 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:14.431338 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp"] Nov 03 16:42:19.412918 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.412875 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" podUID="c834d700-cbe1-4ff6-99f1-93481ac6864f" containerName="payload-processing" containerID="cri-o://d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea" gracePeriod=25 Nov 03 16:42:19.432126 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.432092 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" podUID="72f6517a-baa9-455b-baa7-0cf94a4e82a5" containerName="payload-pre-processing" containerID="cri-o://a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf" gracePeriod=25 Nov 03 16:42:19.682025 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.681975 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:42:19.715939 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.715909 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:42:19.749652 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.749623 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume\") pod \"c834d700-cbe1-4ff6-99f1-93481ac6864f\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " Nov 03 16:42:19.749839 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.749689 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567\" (UniqueName: \"kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567\") pod \"c834d700-cbe1-4ff6-99f1-93481ac6864f\" (UID: \"c834d700-cbe1-4ff6-99f1-93481ac6864f\") " Nov 03 16:42:19.750076 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.750051 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume" pod "c834d700-cbe1-4ff6-99f1-93481ac6864f" (UID: "c834d700-cbe1-4ff6-99f1-93481ac6864f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:19.752061 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.752032 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567" pod "c834d700-cbe1-4ff6-99f1-93481ac6864f" (UID: "c834d700-cbe1-4ff6-99f1-93481ac6864f"). InnerVolumeSpecName "kube-api-access-qk567". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:19.851079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.851039 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume\") pod \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " Nov 03 16:42:19.851280 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.851103 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln\" (UniqueName: \"kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln\") pod \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\" (UID: \"72f6517a-baa9-455b-baa7-0cf94a4e82a5\") " Nov 03 16:42:19.851362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.851318 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c834d700-cbe1-4ff6-99f1-93481ac6864f-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:19.851362 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.851331 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk567\" (UniqueName: \"kubernetes.io/projected/c834d700-cbe1-4ff6-99f1-93481ac6864f-kube-api-access-qk567\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:19.851474 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.851417 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume" pod "72f6517a-baa9-455b-baa7-0cf94a4e82a5" (UID: "72f6517a-baa9-455b-baa7-0cf94a4e82a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:19.853481 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.853457 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln" pod "72f6517a-baa9-455b-baa7-0cf94a4e82a5" (UID: "72f6517a-baa9-455b-baa7-0cf94a4e82a5"). InnerVolumeSpecName "kube-api-access-t99ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:19.952375 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.952258 6319 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f6517a-baa9-455b-baa7-0cf94a4e82a5-plugins-config-volume\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:19.952375 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:19.952324 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t99ln\" (UniqueName: \"kubernetes.io/projected/72f6517a-baa9-455b-baa7-0cf94a4e82a5-kube-api-access-t99ln\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:20.356708 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.356673 6319 generic.go:356] "Generic (PLEG): container finished" podID="72f6517a-baa9-455b-baa7-0cf94a4e82a5" containerID="a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf" exitCode=0 Nov 03 16:42:20.356909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.356726 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" Nov 03 16:42:20.356909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.356746 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" event={"ID":"72f6517a-baa9-455b-baa7-0cf94a4e82a5","Type":"ContainerDied","Data":"a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf"} Nov 03 16:42:20.356909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.356804 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp" event={"ID":"72f6517a-baa9-455b-baa7-0cf94a4e82a5","Type":"ContainerDied","Data":"3e88232569bb26cc7180e9e2903c9a3d2d2f06681634ab8b664ce1c86edc0035"} Nov 03 16:42:20.356909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.356827 6319 scope.go:122] "RemoveContainer" containerID="a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf" Nov 03 16:42:20.358384 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.358360 6319 generic.go:356] "Generic (PLEG): container finished" podID="c834d700-cbe1-4ff6-99f1-93481ac6864f" containerID="d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea" exitCode=0 Nov 03 16:42:20.358516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.358400 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" Nov 03 16:42:20.358516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.358440 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" event={"ID":"c834d700-cbe1-4ff6-99f1-93481ac6864f","Type":"ContainerDied","Data":"d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea"} Nov 03 16:42:20.358516 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.358477 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs" event={"ID":"c834d700-cbe1-4ff6-99f1-93481ac6864f","Type":"ContainerDied","Data":"40f26abe20c91ae8878fda1cdd6bf431099463887494f410a0fbcbc0922096b2"} Nov 03 16:42:20.369085 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.369067 6319 scope.go:122] "RemoveContainer" containerID="a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf" Nov 03 16:42:20.369373 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:42:20.369350 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf\": container with ID starting with a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf not found: ID does not exist" containerID="a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf" Nov 03 16:42:20.369464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.369379 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf"} err="failed to get container status \"a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf\": rpc error: code = NotFound desc = could not find container \"a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf\": container with ID starting with a56a9c9c09d0c25884f28f1083cef8e71b8d6245e17d7efa740723a1a732e9bf not found: ID does not exist" Nov 03 16:42:20.369464 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.369395 6319 scope.go:122] "RemoveContainer" containerID="d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea" Nov 03 16:42:20.379877 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.379853 6319 scope.go:122] "RemoveContainer" containerID="d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea" Nov 03 16:42:20.380258 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:42:20.380234 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea\": container with ID starting with d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea not found: ID does not exist" containerID="d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea" Nov 03 16:42:20.380373 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.380265 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea"} err="failed to get container status \"d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea\": rpc error: code = NotFound desc = could not find container \"d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea\": container with ID starting with d17b73d3bbf03a21f36eca51cae2e17a3a71fb217a2c7c00dcae1673818063ea not found: ID does not exist" Nov 03 16:42:20.389182 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.389158 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp"] Nov 03 16:42:20.400074 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.395590 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-05bcf4-9f5c8795-bgwtp"] Nov 03 16:42:20.414134 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.414107 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs"] Nov 03 16:42:20.441715 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.441689 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-05bcf4-7b48777b79-slfbs"] Nov 03 16:42:20.641742 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.641666 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f6517a-baa9-455b-baa7-0cf94a4e82a5" path="/var/lib/kubelet/pods/72f6517a-baa9-455b-baa7-0cf94a4e82a5/volumes" Nov 03 16:42:20.642071 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:20.642058 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c834d700-cbe1-4ff6-99f1-93481ac6864f" path="/var/lib/kubelet/pods/c834d700-cbe1-4ff6-99f1-93481ac6864f/volumes" Nov 03 16:42:27.771399 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:42:27.771305 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:42:27.771892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:27.771531 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56"] Nov 03 16:42:27.771892 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:27.771784 6319 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" containerID="cri-o://614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0" gracePeriod=30 Nov 03 16:42:33.082714 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.082677 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:42:33.100757 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100729 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.100941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100791 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.100941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100813 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.100941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100842 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.100941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100863 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.100941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100887 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.100941 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100939 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.101303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.100972 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.101303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101066 6319 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data\") pod \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\" (UID: \"aaab4a89-6e52-4544-a1ce-825fcb7f2e85\") " Nov 03 16:42:33.101303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101068 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:33.101303 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101264 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:33.101527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101411 6319 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-certs\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.101527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101432 6319 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-credential-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.101527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101442 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:33.101527 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101488 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:33.101727 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.101557 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:33.103390 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.103361 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:33.103498 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.103394 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:42:33.103805 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.103789 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:33.103848 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.103809 6319 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7" pod "aaab4a89-6e52-4544-a1ce-825fcb7f2e85" (UID: "aaab4a89-6e52-4544-a1ce-825fcb7f2e85"). InnerVolumeSpecName "kube-api-access-n59m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:33.202581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202540 6319 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-token\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.202581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202572 6319 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-envoy\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.202581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202581 6319 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-workload-socket\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.202581 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202591 6319 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59m7\" (UniqueName: \"kubernetes.io/projected/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-kube-api-access-n59m7\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.202859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202601 6319 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istiod-ca-cert\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.202859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202610 6319 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-podinfo\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.202859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.202618 6319 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab4a89-6e52-4544-a1ce-825fcb7f2e85-istio-data\") on node \"ip-10-0-129-122.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 16:42:33.460873 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.460836 6319 generic.go:356] "Generic (PLEG): container finished" podID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerID="614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0" exitCode=0 Nov 03 16:42:33.461081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.460884 6319 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" Nov 03 16:42:33.461081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.460912 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" event={"ID":"aaab4a89-6e52-4544-a1ce-825fcb7f2e85","Type":"ContainerDied","Data":"614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0"} Nov 03 16:42:33.461081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.460973 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56" event={"ID":"aaab4a89-6e52-4544-a1ce-825fcb7f2e85","Type":"ContainerDied","Data":"d7450e3471adb36cb73878b715f841f3c8c2bc3b3ac1c899e0b287c0b663bf92"} Nov 03 16:42:33.461081 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.461011 6319 scope.go:122] "RemoveContainer" containerID="614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0" Nov 03 16:42:33.471428 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.471412 6319 scope.go:122] "RemoveContainer" containerID="3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69" Nov 03 16:42:33.481454 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.481436 6319 scope.go:122] "RemoveContainer" containerID="614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0" Nov 03 16:42:33.481754 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:42:33.481736 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0\": container with ID starting with 614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0 not found: ID does not exist" containerID="614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0" Nov 03 16:42:33.481810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.481761 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0"} err="failed to get container status \"614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0\": rpc error: code = NotFound desc = could not find container \"614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0\": container with ID starting with 614c14b80a941ace37eda075341db01d0de517133e3fc9a5e10b748f1a3240f0 not found: ID does not exist" Nov 03 16:42:33.481810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.481779 6319 scope.go:122] "RemoveContainer" containerID="3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69" Nov 03 16:42:33.482050 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:42:33.482035 6319 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69\": container with ID starting with 3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69 not found: ID does not exist" containerID="3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69" Nov 03 16:42:33.482103 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.482054 6319 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69"} err="failed to get container status \"3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69\": rpc error: code = NotFound desc = could not find container \"3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69\": container with ID starting with 3a0505ed02ff4790d82186e19b92fad7351431cb78c2c644f38cd4c8ddcc2a69 not found: ID does not exist" Nov 03 16:42:33.486925 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.486893 6319 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56"] Nov 03 16:42:33.496597 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:33.496574 6319 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-05bcf4-openshift-default-c9b7f4597-vkm56"] Nov 03 16:42:34.641720 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:42:34.641685 6319 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" path="/var/lib/kubelet/pods/aaab4a89-6e52-4544-a1ce-825fcb7f2e85/volumes" Nov 03 16:42:57.761410 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:42:57.761371 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:43:21.624558 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:21.624524 6319 scope.go:122] "RemoveContainer" containerID="d0310ae77f83e170c921c46471937969aa0fde606a3d3064b1cc6a9a4f18a2a4" Nov 03 16:43:27.761713 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:43:27.761670 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:43:28.696072 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:28.696037 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8c5c5b46c-bg6dh_6793402c-af75-49c1-a697-b0da04a6336f/manager/0.log" Nov 03 16:43:28.886420 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:28.886390 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ntj89_7a4827a0-953e-4290-8c36-a33fe77091bd/manager/0.log" Nov 03 16:43:29.660636 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:29.660603 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4b666674-b7hjc_e15c5062-14a5-4166-a0bb-f63d64a7b792/manager/0.log" Nov 03 16:43:29.843810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:29.843780 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j7zvw_440b9503-cf4d-4749-94b7-eb4101c00c59/server/0.log" Nov 03 16:43:30.042810 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:30.042781 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bjn8p_b78b08a9-11af-446a-abce-551fc42a58e6/manager/0.log" Nov 03 16:43:30.269568 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:30.269538 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4hdlb_a2dceca0-2a1a-461a-9f2b-7384c3109fca/proxy/0.log" Nov 03 16:43:31.238381 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:31.238345 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55668dcc6b-c4ncz_e098d0c5-4c68-48bf-8651-8b2bbe5612f4/maas-api/0.log" Nov 03 16:43:32.515623 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:32.515586 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-b7xr7_4fc056e1-e7f2-44d9-8350-bcd37e9e65e7/postgres/0.log" Nov 03 16:43:34.617218 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:34.617192 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tlqlg_89ed5768-10fd-4495-8be7-559b43295d93/kuadrant-console-plugin/0.log" Nov 03 16:43:36.232329 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:36.232298 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8ggl7_ace6aed5-9f03-485c-84cc-fcf08239b44a/discovery/0.log" Nov 03 16:43:36.585187 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:36.585099 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-8lnw6_244e1060-bcfe-466b-89cf-e85f9e1429a0/kube-auth-proxy/0.log" Nov 03 16:43:36.763254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:36.763225 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-5lcmq_60a875f2-059f-4327-a602-f421a3b2024b/istio-proxy/0.log" Nov 03 16:43:37.338954 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:37.338917 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/1.log" Nov 03 16:43:37.902512 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:37.902482 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/1.log" Nov 03 16:43:38.811333 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:38.811294 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-l4v5g_7c408781-ebf6-49f4-bab3-5e368dc99aab/payload-pre-processing/0.log" Nov 03 16:43:38.996549 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:38.996513 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:43:40.017580 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:40.017548 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/0.log" Nov 03 16:43:40.200491 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:40.200464 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/0.log" Nov 03 16:43:41.119593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:41.119568 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj_cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b/main/0.log" Nov 03 16:43:41.650641 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:41.650612 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-ggwbh_c79ffcfe-b586-4ebf-a29e-f2f6c155784c/main/0.log" Nov 03 16:43:42.023948 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:42.023894 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-l4d5b_bed9f35a-eb7a-4d7d-b328-b5fa380c8676/main/0.log" Nov 03 16:43:42.202532 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:42.202505 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg_634e1091-c4df-49da-9b69-037e924c624e/main/0.log" Nov 03 16:43:52.611467 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:52.611428 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8c5c5b46c-bg6dh_6793402c-af75-49c1-a697-b0da04a6336f/manager/0.log" Nov 03 16:43:52.806270 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:52.806241 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ntj89_7a4827a0-953e-4290-8c36-a33fe77091bd/manager/0.log" Nov 03 16:43:53.424257 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:53.424192 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4b666674-b7hjc_e15c5062-14a5-4166-a0bb-f63d64a7b792/manager/0.log" Nov 03 16:43:53.636386 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:53.636358 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-j7zvw_440b9503-cf4d-4749-94b7-eb4101c00c59/server/0.log" Nov 03 16:43:53.829344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:53.829298 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bjn8p_b78b08a9-11af-446a-abce-551fc42a58e6/manager/0.log" Nov 03 16:43:54.040839 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:54.040810 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4hdlb_a2dceca0-2a1a-461a-9f2b-7384c3109fca/proxy/0.log" Nov 03 16:43:54.776069 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:54.776012 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55668dcc6b-c4ncz_e098d0c5-4c68-48bf-8651-8b2bbe5612f4/maas-api/0.log" Nov 03 16:43:56.051115 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:56.051038 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-b7xr7_4fc056e1-e7f2-44d9-8350-bcd37e9e65e7/postgres/0.log" Nov 03 16:43:57.761801 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:43:57.761764 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:43:57.801811 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:57.801783 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tlqlg_89ed5768-10fd-4495-8be7-559b43295d93/kuadrant-console-plugin/0.log" Nov 03 16:43:59.239795 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:59.239761 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8ggl7_ace6aed5-9f03-485c-84cc-fcf08239b44a/discovery/0.log" Nov 03 16:43:59.595249 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:59.595144 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-8lnw6_244e1060-bcfe-466b-89cf-e85f9e1429a0/kube-auth-proxy/0.log" Nov 03 16:43:59.782430 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:43:59.782393 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-5lcmq_60a875f2-059f-4327-a602-f421a3b2024b/istio-proxy/0.log" Nov 03 16:44:00.362717 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:00.362689 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/1.log" Nov 03 16:44:00.915786 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:00.915756 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/1.log" Nov 03 16:44:01.834272 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:01.834231 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-l4v5g_7c408781-ebf6-49f4-bab3-5e368dc99aab/payload-pre-processing/0.log" Nov 03 16:44:02.008876 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:02.008845 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:44:02.713092 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:02.713062 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wvppj_cb83d6ed-7e19-4b37-b3eb-fe24f40a5f4b/main/0.log" Nov 03 16:44:03.268881 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:03.268848 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-ggwbh_c79ffcfe-b586-4ebf-a29e-f2f6c155784c/main/0.log" Nov 03 16:44:03.652903 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:03.652823 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-l4d5b_bed9f35a-eb7a-4d7d-b328-b5fa380c8676/main/0.log" Nov 03 16:44:03.818836 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:03.818801 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7ffgg_634e1091-c4df-49da-9b69-037e924c624e/main/0.log" Nov 03 16:44:27.762606 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:44:27.762567 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:44:41.286633 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:41.286594 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4b666674-b7hjc_e15c5062-14a5-4166-a0bb-f63d64a7b792/manager/0.log" Nov 03 16:44:41.589893 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:41.589814 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4b666674-b7hjc_e15c5062-14a5-4166-a0bb-f63d64a7b792/manager/0.log" Nov 03 16:44:42.281325 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:42.281294 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:44:42.444665 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:44:42.444629 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:44:57.761832 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:44:57.761796 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:45:27.761740 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:45:27.761664 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:45:28.715904 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:28.715869 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4b666674-b7hjc_e15c5062-14a5-4166-a0bb-f63d64a7b792/manager/0.log" Nov 03 16:45:28.980232 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:28.980149 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c4b666674-b7hjc_e15c5062-14a5-4166-a0bb-f63d64a7b792/manager/0.log" Nov 03 16:45:29.685263 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:29.685233 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:45:29.863736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:29.863702 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:45:33.714740 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:33.714707 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v5zrn_6df99de4-80b2-4cde-b4e8-aa117c1d00de/global-pull-secret-syncer/0.log" Nov 03 16:45:33.972637 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:33.972546 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xmqfv_3eab8211-6bb8-4c65-a1a1-f2d0474952a2/konnectivity-agent/0.log" Nov 03 16:45:33.999078 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:33.999053 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-122.us-east-2.compute.internal_d723d695ab9475e224b694e4eda0376e/haproxy/0.log" Nov 03 16:45:41.336312 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:41.336281 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tlqlg_89ed5768-10fd-4495-8be7-559b43295d93/kuadrant-console-plugin/0.log" Nov 03 16:45:42.830418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:42.830385 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/alertmanager/0.log" Nov 03 16:45:42.868393 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:42.868370 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/config-reloader/0.log" Nov 03 16:45:42.896211 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:42.896186 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/kube-rbac-proxy-web/0.log" Nov 03 16:45:42.934556 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:42.934534 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/kube-rbac-proxy/0.log" Nov 03 16:45:42.964018 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:42.963974 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/kube-rbac-proxy-metric/0.log" Nov 03 16:45:42.992974 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:42.992948 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/prom-label-proxy/0.log" Nov 03 16:45:43.020557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.020536 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00089217-c01a-415e-beb1-b8abc530eb75/init-config-reloader/0.log" Nov 03 16:45:43.242545 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.242515 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-b64r4_d1ae5215-0dc9-4cb1-a88d-1e637fb6ec13/monitoring-plugin/0.log" Nov 03 16:45:43.384593 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.384559 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-snml5_35065e4f-a8be-48cb-97ee-d3fb18c6117e/node-exporter/0.log" Nov 03 16:45:43.413910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.413888 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-snml5_35065e4f-a8be-48cb-97ee-d3fb18c6117e/kube-rbac-proxy/0.log" Nov 03 16:45:43.442084 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.442055 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-snml5_35065e4f-a8be-48cb-97ee-d3fb18c6117e/init-textfile/0.log" Nov 03 16:45:43.560565 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.560485 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ls778_0221604f-4dae-4b2d-8a42-76d243fa48af/kube-rbac-proxy-main/0.log" Nov 03 16:45:43.587866 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.587831 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ls778_0221604f-4dae-4b2d-8a42-76d243fa48af/kube-rbac-proxy-self/0.log" Nov 03 16:45:43.613899 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.613869 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ls778_0221604f-4dae-4b2d-8a42-76d243fa48af/openshift-state-metrics/0.log" Nov 03 16:45:43.664859 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.664827 6319 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz"] Nov 03 16:45:43.665254 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665241 6319 cpu_manager.go:414] "containerMap: removing container" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665256 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665269 6319 cpu_manager.go:414] "containerMap: removing container" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665275 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665290 6319 cpu_manager.go:414] "containerMap: removing container" podUID="72f6517a-baa9-455b-baa7-0cf94a4e82a5" containerName="payload-pre-processing" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665295 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f6517a-baa9-455b-baa7-0cf94a4e82a5" containerName="payload-pre-processing" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665312 6319 cpu_manager.go:414] "containerMap: removing container" podUID="c834d700-cbe1-4ff6-99f1-93481ac6864f" containerName="payload-processing" Nov 03 16:45:43.665326 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665316 6319 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c834d700-cbe1-4ff6-99f1-93481ac6864f" containerName="payload-processing" Nov 03 16:45:43.665576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665385 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" Nov 03 16:45:43.665576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665394 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaab4a89-6e52-4544-a1ce-825fcb7f2e85" containerName="istio-proxy" Nov 03 16:45:43.665576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665401 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f6517a-baa9-455b-baa7-0cf94a4e82a5" containerName="payload-pre-processing" Nov 03 16:45:43.665576 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.665408 6319 memory_manager.go:365] "RemoveStaleState removing state" podUID="c834d700-cbe1-4ff6-99f1-93481ac6864f" containerName="payload-processing" Nov 03 16:45:43.668340 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.668320 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.676835 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.676706 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2xlg\"/\"openshift-service-ca.crt\"" Nov 03 16:45:43.676835 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.676809 6319 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2xlg\"/\"kube-root-ca.crt\"" Nov 03 16:45:43.678113 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.678095 6319 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v2xlg\"/\"default-dockercfg-qmzbm\"" Nov 03 16:45:43.679872 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.679853 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz"] Nov 03 16:45:43.781127 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.781094 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-podres\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.781127 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.781136 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-sys\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.781344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.781169 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbdz\" (UniqueName: \"kubernetes.io/projected/099434a8-e1e4-41fb-87d8-75cd4327de78-kube-api-access-kdbdz\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.781344 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.781311 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-lib-modules\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.781458 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.781347 6319 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-proc\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.881828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881750 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-lib-modules\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.881828 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881794 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-proc\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881834 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-podres\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881849 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-sys\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881871 6319 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbdz\" (UniqueName: \"kubernetes.io/projected/099434a8-e1e4-41fb-87d8-75cd4327de78-kube-api-access-kdbdz\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881907 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-lib-modules\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881958 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-podres\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881970 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-proc\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.882273 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.881977 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/099434a8-e1e4-41fb-87d8-75cd4327de78-sys\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.894736 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.894715 6319 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbdz\" (UniqueName: \"kubernetes.io/projected/099434a8-e1e4-41fb-87d8-75cd4327de78-kube-api-access-kdbdz\") pod \"perf-node-gather-daemonset-l4xxz\" (UID: \"099434a8-e1e4-41fb-87d8-75cd4327de78\") " pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:43.982100 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:43.982072 6319 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:44.033537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.032082 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6b649c8c-qntdf_9523fdc9-2d03-4d32-9a63-26b5246ed5a0/telemeter-client/0.log" Nov 03 16:45:44.059621 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.059512 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6b649c8c-qntdf_9523fdc9-2d03-4d32-9a63-26b5246ed5a0/reload/0.log" Nov 03 16:45:44.090862 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.090840 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6b649c8c-qntdf_9523fdc9-2d03-4d32-9a63-26b5246ed5a0/kube-rbac-proxy/0.log" Nov 03 16:45:44.259910 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.259887 6319 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz"] Nov 03 16:45:44.262091 ip-10-0-129-122 kubenswrapper[6319]: W1103 16:45:44.262063 6319 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod099434a8_e1e4_41fb_87d8_75cd4327de78.slice/crio-52c656fa9749bc836ac0a949d017d59c105521ab0a3dfb8c35f5f13db5ff8162 WatchSource:0}: Error finding container 52c656fa9749bc836ac0a949d017d59c105521ab0a3dfb8c35f5f13db5ff8162: Status 404 returned error can't find the container with id 52c656fa9749bc836ac0a949d017d59c105521ab0a3dfb8c35f5f13db5ff8162 Nov 03 16:45:44.426079 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.426042 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" event={"ID":"099434a8-e1e4-41fb-87d8-75cd4327de78","Type":"ContainerStarted","Data":"5635c62206686e2c8f82b85f79053309b2739cb80034e85c3190f0fcc8150bef"} Nov 03 16:45:44.426247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.426102 6319 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" event={"ID":"099434a8-e1e4-41fb-87d8-75cd4327de78","Type":"ContainerStarted","Data":"52c656fa9749bc836ac0a949d017d59c105521ab0a3dfb8c35f5f13db5ff8162"} Nov 03 16:45:44.426247 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.426126 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:44.445537 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:44.445491 6319 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" podStartSLOduration=1.445477412 podStartE2EDuration="1.445477412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:44.444875736 +0000 UTC m=+2547.956243606" watchObservedRunningTime="2026-11-03 16:45:44.445477412 +0000 UTC m=+2547.956845297" Nov 03 16:45:46.584598 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:46.584567 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-565565c4b5-zqhq2_c8b2287e-d813-4fb2-bea3-5443013af667/console/0.log" Nov 03 16:45:48.531161 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:48.531130 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jksqm_1ee13f98-11c4-4b25-87e1-f0c4729750f9/dns/0.log" Nov 03 16:45:48.555418 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:48.555384 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jksqm_1ee13f98-11c4-4b25-87e1-f0c4729750f9/kube-rbac-proxy/0.log" Nov 03 16:45:48.699908 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:48.699878 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgtnn_7128ff3f-3a5a-40c8-98a1-2ec63572519b/dns-node-resolver/0.log" Nov 03 16:45:49.387998 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:49.387963 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5726_7f7b62ab-f1df-47f3-99b3-b34efd73a23e/node-ca/0.log" Nov 03 16:45:50.440672 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:50.440640 6319 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v2xlg/perf-node-gather-daemonset-l4xxz" Nov 03 16:45:52.769719 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:52.769685 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8ggl7_ace6aed5-9f03-485c-84cc-fcf08239b44a/discovery/0.log" Nov 03 16:45:52.852219 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:52.852183 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-8lnw6_244e1060-bcfe-466b-89cf-e85f9e1429a0/kube-auth-proxy/0.log" Nov 03 16:45:52.887180 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:52.887148 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-5lcmq_60a875f2-059f-4327-a602-f421a3b2024b/istio-proxy/0.log" Nov 03 16:45:52.996438 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:52.996403 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/1.log" Nov 03 16:45:52.996784 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:52.996763 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/0.log" Nov 03 16:45:53.118767 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:53.118683 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/0.log" Nov 03 16:45:53.119121 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:53.119098 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/1.log" Nov 03 16:45:53.300305 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:53.300277 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-l4v5g_7c408781-ebf6-49f4-bab3-5e368dc99aab/payload-pre-processing/0.log" Nov 03 16:45:53.334643 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:53.334613 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:45:54.868166 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:54.868124 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6tc5s_e678d92f-19e7-49f2-b876-774d845ac141/serve-healthcheck-canary/0.log" Nov 03 16:45:55.500587 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:55.500550 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q4t6_2a3725d4-d55d-42eb-896f-9d10f59cb5ef/kube-rbac-proxy/0.log" Nov 03 16:45:55.529241 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:55.529215 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q4t6_2a3725d4-d55d-42eb-896f-9d10f59cb5ef/exporter/0.log" Nov 03 16:45:55.559330 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:55.559301 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q4t6_2a3725d4-d55d-42eb-896f-9d10f59cb5ef/extractor/0.log" Nov 03 16:45:56.518844 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.518783 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8ggl7_ace6aed5-9f03-485c-84cc-fcf08239b44a/discovery/0.log" Nov 03 16:45:56.557781 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.557753 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dbfdb86-8lnw6_244e1060-bcfe-466b-89cf-e85f9e1429a0/kube-auth-proxy/0.log" Nov 03 16:45:56.585421 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.585390 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-5lcmq_60a875f2-059f-4327-a602-f421a3b2024b/istio-proxy/0.log" Nov 03 16:45:56.642197 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.642167 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/1.log" Nov 03 16:45:56.642557 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.642539 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-ldj54_8c35b1f9-9955-47e5-9864-6ff55045a21a/istio-proxy/0.log" Nov 03 16:45:56.700909 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.700884 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/0.log" Nov 03 16:45:56.701268 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.701250 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-wlqrl_af06ee06-d105-42d0-8efc-77603192cc29/istio-proxy/1.log" Nov 03 16:45:56.791041 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.790968 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-l4v5g_7c408781-ebf6-49f4-bab3-5e368dc99aab/payload-pre-processing/0.log" Nov 03 16:45:56.823508 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:56.823483 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-87krw_5179a7ba-833e-4b8b-9ce5-f2b5a77e0253/payload-processing/0.log" Nov 03 16:45:57.761536 ip-10-0-129-122 kubenswrapper[6319]: E1103 16:45:57.761500 6319 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4q4t6" containerName="extractor" Nov 03 16:45:58.268808 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:58.268764 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-d59cb86d5-46fvd_b4fddbce-22a2-40c8-876a-3f5baeb36b4f/manager/0.log" Nov 03 16:45:58.299357 ip-10-0-129-122 kubenswrapper[6319]: I1103 16:45:58.299333 6319 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-d59cb86d5-46fvd_b4fddbce-22a2-40c8-876a-3f5baeb36b4f/copy-manifests/0.log"